Oct 30 05:52:30.740618 ip-10-0-136-123 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:52:31.190940 ip-10-0-136-123 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:52:31.190940 ip-10-0-136-123 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:52:31.190940 ip-10-0-136-123 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:52:31.190940 ip-10-0-136-123 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:52:31.195545 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195463 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:31.195545 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195541 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:31.195545 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195546 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:31.195545 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195549 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:31.195545 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195553 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195558 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195562 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195565 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195569 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195573 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195577 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195580 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195586 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195590 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195594 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195599 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195602 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195606 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195610 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195616 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195620 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195624 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195628 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:31.195830 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195632 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195636 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195640 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195644 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195648 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195651 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195655 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195661 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195667 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195673 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195678 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195683 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195687 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195691 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195696 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195701 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195706 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195710 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195714 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:31.196626 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195718 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195723 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195727 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195732 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195739 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195745 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195750 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195756 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195760 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195765 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195771 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195775 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195780 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195784 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195788 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195792 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195796 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195801 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195805 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195809 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:31.197446 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195813 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195818 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195822 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195826 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195829 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195834 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195838 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195842 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195847 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195851 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195856 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195860 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195864 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195869 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195873 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195877 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195883 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195887 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195891 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:31.198273 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195896 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195900 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195905 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195909 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195915 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195919 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195924 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195928 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195932 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195936 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195940 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195945 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195949 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195953 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195958 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195962 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195967 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195971 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195975 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195981 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:31.199139 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195988 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195992 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.195996 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.196001 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.196005 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197281 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197290 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197294 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197316 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197320 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197324 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197329 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197333 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197337 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197342 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197346 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197350 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197355 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197359 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197363 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:31.199647 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197368 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197373 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197377 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197382 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197389 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197395 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197399 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197404 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197408 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197413 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197417 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197421 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197426 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197430 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197435 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197442 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197448 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197452 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:31.200328 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197456 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197460 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197464 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197468 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197474 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197478 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197482 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197486 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197490 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197495 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197499 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197503 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197508 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197512 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197516 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197520 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197524 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197530 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197534 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197538 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:31.201095 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197542 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197547 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197551 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197555 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197559 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197563 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197567 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197571 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197575 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197579 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197583 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197587 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197592 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197596 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197600 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197604 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197608 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197612 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197617 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197621 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:31.201905 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197625 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197629 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197633 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197637 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197641 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197646 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197651 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197655 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197659 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197664 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197668 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197672 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197676 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197680 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197684 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197689 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197692 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197696 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197701 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197705 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:31.202663 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197709 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197713 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197717 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197721 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197726 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197730 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197734 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197739 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197743 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197747 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197751 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197755 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.197761 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198560 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198569 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198573 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198578 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198583 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198588 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198593 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:31.203409 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198598 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198603 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198610 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198615 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198620 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198625 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198629 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198633 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198637 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198641 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198645 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198650 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198654 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198658 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198663 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198667 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198671 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198675 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198679 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:31.204584 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198688 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198692 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198697 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198701 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198705 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198710 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198714 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198718 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198722 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198726 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198730 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198735 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198739 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198744 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198748 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198752 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198756 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198761 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198765 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198769 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:31.205031 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198772 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198777 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198781 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198785 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198789 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198793 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198797 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198801 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198805 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198812 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198817 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198821 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198827 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198831 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198835 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198839 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198843 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198847 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198852 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:31.205538 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198856 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198860 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198864 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198868 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198872 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198877 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198883 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198887 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198891 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198895 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198899 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198903 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198909 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198914 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198920 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198924 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198929 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198933 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198937 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198941 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:31.205980 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198945 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198949 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198955 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198959 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198964 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198969 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198973 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198978 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198983 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198987 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198991 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.198995 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.199000 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.199004 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.199008 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.199012 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.199016 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.199020 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.199025 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:31.206464 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.199029 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.199033 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200002 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200017 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200025 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200033 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200039 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200045 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200052 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200059 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200064 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200069 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200075 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200080 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200085 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200093 2573 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200098 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200104 2573 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200111 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200116 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200125 2573 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200129 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200135 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200140 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:52:31.206924 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200145 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200151 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200156 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200161 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200167 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200171 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200176 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200181 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200187 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200192 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200199 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200204 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200209 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200214 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200219 2573 flags.go:64] FLAG: --enable-server="true" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200224 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200230 2573 flags.go:64] FLAG: --event-burst="100" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200234 2573 flags.go:64] FLAG: --event-qps="50" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200239 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200244 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200249 2573 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200256 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200261 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200267 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200272 2573 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:52:31.207500 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200277 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200284 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200288 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200293 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200313 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200319 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200324 2573 flags.go:64] FLAG: --feature-gates="" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200330 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200335 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200339 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200344 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200349 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200354 2573 flags.go:64] FLAG: --help="false" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200359 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200363 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200368 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200374 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200380 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200385 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200390 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200395 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200399 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200404 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:52:31.208197 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200408 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200414 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200418 2573 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200423 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200428 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200433 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200437 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200442 2573 flags.go:64] FLAG: --lock-file="" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200447 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200451 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200458 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200467 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200472 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200476 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200481 2573 flags.go:64] FLAG: --logging-format="text" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200485 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200491 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200495 2573 flags.go:64] FLAG: --manifest-url="" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200500 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200507 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200512 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200518 2573 flags.go:64] FLAG: --max-pods="110" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200522 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200527 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200532 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:52:31.208774 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200536 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200543 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200547 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200552 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200563 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200567 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200572 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200577 2573 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200581 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200586 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200591 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200596 2573 flags.go:64] FLAG: --port="10250" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200607 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200612 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b77122a4542ef9cd" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200617 2573 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200622 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200627 2573 flags.go:64] FLAG: --register-node="true" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200633 2573 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200640 2573 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200645 2573 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200649 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200654 2573 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200660 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200664 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200669 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200674 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:52:31.209440 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200679 2573 flags.go:64] FLAG: --runonce="false" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200684 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200689 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200694 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200698 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200703 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200708 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200713 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200718 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200723 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200728 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200733 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200738 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200743 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200748 2573 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200752 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200761 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200766 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200770 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200777 2573 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200782 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200786 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200791 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200797 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200802 2573 flags.go:64] FLAG: --v="2" Oct 30 05:52:31.210043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200809 2573 flags.go:64] FLAG: --version="false" Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200815 2573 flags.go:64] FLAG: --vmodule="" Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200822 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.200827 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203657 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203676 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203680 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203684 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203687 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203690 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203694 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203697 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203699 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203702 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203705 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203708 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203710 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203713 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203716 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203718 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:31.210654 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203721 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203724 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203726 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203733 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203742 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203745 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203748 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203751 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203754 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203757 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203760 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203763 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203765 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203768 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203771 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203773 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203776 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203779 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:31.211128 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203782 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203784 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203787 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203789 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203792 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203795 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203798 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203800 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203803 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203806 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203808 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203811 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203813 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203816 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203818 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203821 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203823 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203827 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203829 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203832 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:31.211611 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203835 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203837 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203840 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203843 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203846 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203848 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203851 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203854 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203857 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203860 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203863 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203865 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203868 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203871 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203873 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203876 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203879 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203881 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203883 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203886 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:31.212123 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203889 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203892 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203894 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203897 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203899 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203902 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203905 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203907 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203911 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203914 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203917 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203919 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203921 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203924 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203927 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203930 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203933 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203935 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203938 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:31.212614 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203940 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203943 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203946 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203949 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203952 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203954 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203957 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203959 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203962 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203966 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203970 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.203973 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.204000 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204041 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204046 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204049 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:31.213050 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204052 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204055 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204058 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204061 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204064 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204066 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204070 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204072 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204075 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204078 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204081 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204083 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204086 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204088 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204091 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204094 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204096 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204099 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204102 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204104 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:31.213427 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204107 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204109 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204112 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204114 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204117 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204120 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204122 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204125 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204127 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204129 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204132 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204135 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204137 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204140 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204142 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204145 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204147 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204150 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204152 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:31.213902 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204155 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204158 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204160 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204163 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204168 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204171 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204173 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204176 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204179 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204181 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204183 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204186 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204190 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204194 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204196 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204200 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204204 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204207 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204210 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:31.214368 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204212 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204215 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204217 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204220 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204222 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204225 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204227 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204230 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204232 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204235 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204238 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204241 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204243 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204246 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204249 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204252 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204255 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204260 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204263 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:31.214814 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204266 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204269 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204271 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204274 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204276 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204279 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204282 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204284 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204287 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204289 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204292 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204294 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204297 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204318 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204322 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204327 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204331 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204339 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204343 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204345 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:31.215261 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204348 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204351 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204353 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204356 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204358 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.204361 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.211495 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.211511 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.212990 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.212998 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213055 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213060 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213063 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213066 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213069 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213072 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213075 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213078 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213081 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:31.215734 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213084 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213087 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213091 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213095 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213098 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213100 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213104 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213106 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213109 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213112 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213114 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213117 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213120 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213123 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213126 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213128 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213131 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213134 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213136 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:31.216177 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213139 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213143 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213150 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213153 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213156 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213159 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213161 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213164 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213167 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213169 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213172 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213174 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213177 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213180 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213182 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213185 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213187 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213190 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213192 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213195 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:31.216637 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213198 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213200 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213203 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213205 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213208 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213210 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213213 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213216 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213218 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213221 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213224 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213226 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213229 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213233 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213236 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213239 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213241 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213244 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213247 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:31.217087 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213249 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213252 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213255 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213257 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213260 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213262 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213265 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213267 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213270 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213272 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213274 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213277 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213280 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213282 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213284 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213289 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213293 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213296 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213308 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:31.217608 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213311 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213314 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213316 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213319 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213322 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213324 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213327 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213330 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213333 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213335 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213338 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213341 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213343 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213346 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213348 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213351 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213354 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213356 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213359 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:31.218048 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213361 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.213390 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213540 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213545 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213548 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213551 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213554 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213556 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213559 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213561 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213564 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213567 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213570 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213572 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213575 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213577 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:31.218507 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213580 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213582 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213586 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213589 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213592 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213595 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213598 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213600 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213603 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213605 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213608 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213611 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213613 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213615 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213618 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213620 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213623 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213626 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213628 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213630 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:31.218879 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213633 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213635 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213638 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213640 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213643 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213645 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213648 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213651 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213653 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213655 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213658 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213662 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213666 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213670 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213673 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213675 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213678 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213680 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:31.219364 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213682 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213685 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213687 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213691 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213693 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213696 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213698 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213701 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213703 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213706 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213708 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213711 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213713 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213716 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213718 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213721 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213723 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213726 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213728 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:31.219779 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213731 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213734 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213736 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213738 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213741 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213743 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213746 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213748 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213750 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213753 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213756 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213758 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213775 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213778 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213780 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213783 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213785 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213788 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213791 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213794 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:31.220226 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213796 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213799 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213801 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213803 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213806 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213809 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213811 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213814 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213816 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213819 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213821 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213824 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213827 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213829 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213832 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.213855 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:52:31.220711 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213889 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213892 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213895 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213898 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213901 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213903 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213906 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213909 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213912 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213915 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213917 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213920 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213923 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213925 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213928 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213931 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213933 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213936 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213938 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213941 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:31.221103 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213943 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213946 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213949 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213951 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213956 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213959 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213962 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213964 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213967 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213970 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213975 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213979 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213981 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213985 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213988 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213991 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213993 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213996 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.213998 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214001 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:31.221607 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214004 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214006 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214009 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214011 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214014 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214016 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214019 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214022 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214024 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214027 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214029 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214032 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214035 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214037 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214040 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214042 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214047 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214050 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214053 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:31.222068 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214057 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214061 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214064 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214067 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214069 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214072 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214075 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214077 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214080 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214083 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214087 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214090 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214092 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214095 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214097 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214100 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214103 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214105 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214108 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:31.222541 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214110 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214113 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214116 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214119 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214121 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214124 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214126 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214129 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214132 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214134 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214138 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214141 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214143 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214146 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214149 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214151 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214154 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214156 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214174 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214178 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:31.222976 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214181 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214185 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214187 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214191 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214194 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214198 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214200 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:31.214203 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.214251 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.216466 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.217461 2573 server.go:1014] "Starting client certificate rotation" Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.217554 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.217596 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.219717 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:52:31.223452 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.220812 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:52:31.235445 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.235430 2573 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:52:31.239101 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.239084 2573 log.go:25] "Validated CRI v1 image API" Oct 30 05:52:31.239875 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.239854 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:52:31.246601 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.246570 2573 fs.go:135] Filesystem UUIDs: map[59820a08-003b-4739-a183-165dd122251e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e7c99e0a-7a8b-4bea-8062-c868a1269592:/dev/nvme0n1p3] Oct 30 05:52:31.246697 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.246600 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 05:52:31.253052 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.252948 2573 manager.go:217] Machine: {Timestamp:2026-10-30 05:52:31.251420158 +0000 UTC m=+0.383212055 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099718 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20894e3385c55d85aec0c625373e30 SystemUUID:ec20894e-3385-c55d-85ae-c0c625373e30 BootID:f602cb3c-e710-4f4a-b171-f8fef0b62d50 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d7:f6:49:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d7:f6:49:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:9b:67:bf:c3:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:52:31.253052 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.253052 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:52:31.253164 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.253121 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:52:31.256817 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.256794 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:52:31.256952 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.256820 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:52:31.256998 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.256954 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:52:31.256998 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.256963 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:52:31.256998 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.256995 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:52:31.257937 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.257927 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:52:31.259161 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.259150 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:52:31.263775 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.263763 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:52:31.263881 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.263872 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:52:31.264856 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.264846 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:52:31.264889 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.264859 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:52:31.264889 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.264870 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:52:31.264889 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.264878 2573 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:52:31.264889 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.264887 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:52:31.265218 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.265203 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:52:31.267641 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.267623 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:52:31.274722 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.274709 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:52:31.275658 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.275646 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:52:31.275703 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.275675 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:52:31.275829 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.275819 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:52:31.275864 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.275835 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:52:31.275864 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.275847 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:52:31.275864 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.275856 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:52:31.275864 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.275862 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:52:31.275962 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.275868 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:52:31.276765 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.276755 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:52:31.276799 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.276766 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:52:31.276799 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.276774 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:52:31.276799 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.276780 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:52:31.276799 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.276794 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:52:31.276992 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.276803 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:52:31.277515 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.277506 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:52:31.277515 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.277516 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:52:31.281216 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.281192 2573 server.go:1263] "Started kubelet" Oct 30 05:52:31.281380 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.281290 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:52:31.281464 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.281398 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:52:31.281839 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.281807 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:52:31.282853 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.282829 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:52:31.284011 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.283998 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:52:31.284388 ip-10-0-136-123 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:52:31.287400 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.287384 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 05:52:31.292749 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.292734 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:52:31.292806 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.292750 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:52:31.292863 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.292821 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:31.293502 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293485 2573 factory.go:55] Registering systemd factory Oct 30 05:52:31.293502 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293505 2573 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:52:31.293656 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293516 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:52:31.293737 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293721 2573 factory.go:153] Registering CRI-O factory Oct 30 05:52:31.293789 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293742 2573 factory.go:223] Registration of the crio container factory successfully Oct 30 05:52:31.293789 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293744 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:52:31.293789 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293756 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:52:31.293912 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293792 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:52:31.293912 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293811 2573 factory.go:103] Registering Raw factory Oct 30 05:52:31.293912 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293826 2573 manager.go:1199] Started watching for new ooms in manager Oct 30 05:52:31.293912 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293867 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:52:31.293912 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.293879 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:52:31.295997 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.295970 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:52:31.296186 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.295985 2573 manager.go:319] Starting recovery of all containers Oct 30 05:52:31.301203 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.301181 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2fjl9" Oct 30 05:52:31.309097 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.308465 2573 manager.go:324] Recovery completed Oct 30 05:52:31.312535 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.312515 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:31.313257 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.313242 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2fjl9" Oct 30 05:52:31.314148 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.314133 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:31.314213 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.314163 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:31.314213 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.314174 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:31.314721 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.314684 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 05:52:31.314721 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.314693 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 05:52:31.314839 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.314800 2573 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:52:31.314839 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.314811 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:52:31.314839 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.314830 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:52:31.314926 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.314812 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:31.315617 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.315599 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:31.315714 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.315625 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:31.315714 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.315636 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:31.315870 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.314683 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-123.ec2.internal.18e338d203c5a299 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-123.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-123.ec2.internal,},FirstTimestamp:2026-10-30 05:52:31.281160857 +0000 UTC m=+0.412952753,LastTimestamp:2026-10-30 05:52:31.281160857 +0000 UTC m=+0.412952753,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-123.ec2.internal,}" Oct 30 05:52:31.317311 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.317285 2573 policy_none.go:50] "Start" Oct 30 05:52:31.317377 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.317316 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:52:31.317377 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.317327 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:52:31.319264 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.319251 2573 policy_none.go:44] "Start" Oct 30 05:52:31.330581 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.330512 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-123.ec2.internal.18e338d205bcf264 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-123.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-123.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-123.ec2.internal,},FirstTimestamp:2026-10-30 05:52:31.314145892 +0000 UTC m=+0.445937790,LastTimestamp:2026-10-30 05:52:31.314145892 +0000 UTC m=+0.445937790,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-123.ec2.internal,}" Oct 30 05:52:31.331531 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.331510 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 05:52:31.331598 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.331507 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 05:52:31.331598 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.331507 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 05:52:31.359868 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.359832 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:52:31.364006 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.363994 2573 manager.go:341] "Starting Device Plugin manager" Oct 30 05:52:31.364069 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.364018 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:52:31.364069 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.364026 2573 server.go:87] "Starting device plugin registration server" Oct 30 05:52:31.364225 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.364215 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:52:31.364260 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.364225 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:52:31.364348 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.364336 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:52:31.364443 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.364427 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:52:31.364443 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.364442 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:52:31.365147 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.364991 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:52:31.365203 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.365178 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:31.434248 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.434217 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:52:31.435719 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.435378 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:52:31.435719 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.435396 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:52:31.435719 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.435412 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:52:31.435719 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.435440 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 05:52:31.447873 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.447831 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:31.464337 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.464310 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:31.465138 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.465124 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:31.465192 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.465151 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:31.465192 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.465160 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:31.465192 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.465181 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.482457 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.482439 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.482509 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.482465 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-123.ec2.internal\": node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:31.536513 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.536477 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"] Oct 30 05:52:31.536610 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.536545 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:31.543156 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.542846 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:31.543156 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.542877 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:31.543156 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.542891 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:31.544792 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.544777 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:31.544927 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.544911 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.544964 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.544945 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:31.545511 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.545495 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:31.545590 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.545521 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:31.545590 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.545530 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:31.545590 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.545499 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:31.545687 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.545596 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:31.545687 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.545614 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:31.547365 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.547352 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.547406 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.547375 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:31.548076 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.548058 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:31.548165 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.548090 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:31.548165 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.548105 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:31.556815 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.556799 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:31.578174 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.578157 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-123.ec2.internal\" not found" node="ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.580908 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.580894 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-123.ec2.internal\" not found" node="ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.596360 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.596342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.596443 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.596367 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.596443 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.596384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc0cc40e9c1a761dcecd1aa2a7591d53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" (UID: \"cc0cc40e9c1a761dcecd1aa2a7591d53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.656876 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.656854 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:31.697366 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.697344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.697475 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.697369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.697475 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.697387 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc0cc40e9c1a761dcecd1aa2a7591d53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" (UID: \"cc0cc40e9c1a761dcecd1aa2a7591d53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.697475 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.697439 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc0cc40e9c1a761dcecd1aa2a7591d53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" (UID: \"cc0cc40e9c1a761dcecd1aa2a7591d53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.697475 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.697451 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.697475 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.697457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.757197 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.757147 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:31.857934 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.857916 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:31.881110 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.881082 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.883931 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:31.883906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:52:31.958223 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:31.958200 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:32.063102 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:32.063049 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:32.167436 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:32.167411 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:32.217316 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.217290 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 05:52:32.217710 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.217451 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 05:52:32.268038 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:32.267991 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:32.319262 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.319179 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:47:31 +0000 UTC" deadline="2028-06-29 10:00:58.449733033 +0000 UTC" Oct 30 05:52:32.319262 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.319224 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14596h8m26.130511597s" Oct 30 05:52:32.368985 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:32.368950 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:32.447074 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:32.447032 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8a05fdf5abb06be0c4a4fee3879794.slice/crio-6d68842197a388956e4be93267daa955a701e20d75801d34e8b4f24a7af19132 WatchSource:0}: Error finding container 6d68842197a388956e4be93267daa955a701e20d75801d34e8b4f24a7af19132: Status 404 returned error can't find the container with id 6d68842197a388956e4be93267daa955a701e20d75801d34e8b4f24a7af19132 Oct 30 05:52:32.447785 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:32.447762 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0cc40e9c1a761dcecd1aa2a7591d53.slice/crio-f549965acef59f07eb50a042f648b8e4c483d466fdb8322854e3e42ac59abfb7 WatchSource:0}: Error finding container f549965acef59f07eb50a042f648b8e4c483d466fdb8322854e3e42ac59abfb7: Status 404 returned error can't find the container with id f549965acef59f07eb50a042f648b8e4c483d466fdb8322854e3e42ac59abfb7 Oct 30 05:52:32.451840 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.451821 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:32.451922 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.451876 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:52:32.453107 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.453084 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:32.453166 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.453132 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:32.453166 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.453149 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:32.469407 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:32.469385 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:32.570502 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:32.570457 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:52:32.610290 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.610270 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:32.653966 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.653942 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:32.693936 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.693905 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:52:32.719629 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.719613 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:52:32.720529 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.720515 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:52:32.743003 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.742989 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:52:32.923894 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:32.923822 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:33.266229 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.266149 2573 apiserver.go:52] "Watching apiserver" Oct 30 05:52:33.293529 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.293497 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 05:52:33.313032 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.313007 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:52:33.313424 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.313390 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-8ntvq","kube-system/konnectivity-agent-kf896","openshift-network-operator/iptables-alerter-cnmml","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj","openshift-cluster-node-tuning-operator/tuned-9qrxj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal","openshift-multus/multus-2fn7x","openshift-dns/node-resolver-b4v22","openshift-multus/multus-additional-cni-plugins-82lcq","openshift-network-diagnostics/network-check-target-2jjjp","openshift-ovn-kubernetes/ovnkube-node-8zb4l","kube-system/global-pull-secret-syncer-lbqgc","kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal","openshift-image-registry/node-ca-v2swz"] Oct 30 05:52:33.315984 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.315963 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.318474 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.318455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.320715 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.320693 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.327574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.327552 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.329881 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.329859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.332727 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.332454 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:33.332727 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.332540 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:33.334621 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.334532 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:33.334621 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.334619 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:33.334868 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.334847 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-27cwd\"" Oct 30 05:52:33.335597 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.335575 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mpzcb\"" Oct 30 05:52:33.335783 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.335768 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:52:33.335878 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.335861 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:52:33.337938 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.337917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:33.338699 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.338678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.340174 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.340158 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:52:33.341262 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.341244 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:33.341371 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.341323 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:33.344027 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.344004 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.347136 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.347111 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:33.347262 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.347184 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:33.350150 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.350125 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.361371 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.361346 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:52:33.361517 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.361425 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qq2cc" Oct 30 05:52:33.395192 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.394963 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:52:33.408761 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.408733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.408868 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.408780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovn-node-metrics-cert\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.408868 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.408811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cni-binary-copy\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.408868 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.408837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-k8s-cni-cncf-io\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.408868 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.408864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-netns\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.409107 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.408889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q88h\" (UniqueName: \"kubernetes.io/projected/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-kube-api-access-7q88h\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.409107 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.408935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-sys-fs\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.409107 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.408959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-netns\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.409107 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a98d05d0-c37f-4a97-b943-fd4b728e29d0-host\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.409107 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409043 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cnibin\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.409107 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkcs\" (UniqueName: \"kubernetes.io/projected/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-kube-api-access-2fkcs\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.409107 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-socket-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.409107 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-log-socket\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409127 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-conf-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xcf5\" (UniqueName: \"kubernetes.io/projected/d2edb037-173f-4c27-987a-72d43df7fafb-kube-api-access-6xcf5\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409228 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-daemon-config\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-agent-certs\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409299 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-sys\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsk8\" (UniqueName: \"kubernetes.io/projected/a98d05d0-c37f-4a97-b943-fd4b728e29d0-kube-api-access-bcsk8\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-conf\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-systemd-units\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-slash\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-var-lib-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-etc-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.409553 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-tuned\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409560 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-system-cni-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-kubelet-config\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409631 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409652 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-lib-modules\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409669 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-tmp\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrk9g\" (UniqueName: \"kubernetes.io/projected/b5705935-3763-404f-a84c-abc0133161bd-kube-api-access-vrk9g\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-socket-dir-parent\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409807 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-dbus\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-node-log\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a98d05d0-c37f-4a97-b943-fd4b728e29d0-serviceca\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.410363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsr4\" (UniqueName: \"kubernetes.io/projected/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-kube-api-access-dhsr4\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5705935-3763-404f-a84c-abc0133161bd-iptables-alerter-script\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-kubernetes\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-var-lib-kubelet\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.409982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9876\" (UniqueName: \"kubernetes.io/projected/bd1a2002-0045-4025-8c34-23ab182ca2a6-kube-api-access-l9876\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-kubelet\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-konnectivity-ca\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-registration-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410143 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-kubelet\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5705935-3763-404f-a84c-abc0133161bd-host-slash\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cnibin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410193 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-multus\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410208 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44xcr\" (UniqueName: \"kubernetes.io/projected/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-kube-api-access-44xcr\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410224 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-env-overrides\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410256 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-bin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-etc-kubernetes\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.411200 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-modprobe-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-run\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-host\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-config\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-script-lib\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-hostroot\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410492 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-multus-certs\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-systemd\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqlk\" (UniqueName: \"kubernetes.io/projected/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-kube-api-access-zzqlk\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-bin\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410677 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-netd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-hosts-file\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-tmp-dir\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-system-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.412033 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysconfig\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.412914 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410833 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-device-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.412914 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-systemd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.412914 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.410884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-ovn\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.440951 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.440832 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" event={"ID":"7c8a05fdf5abb06be0c4a4fee3879794","Type":"ContainerStarted","Data":"6d68842197a388956e4be93267daa955a701e20d75801d34e8b4f24a7af19132"} Oct 30 05:52:33.442861 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.442829 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" event={"ID":"cc0cc40e9c1a761dcecd1aa2a7591d53","Type":"ContainerStarted","Data":"f549965acef59f07eb50a042f648b8e4c483d466fdb8322854e3e42ac59abfb7"} Oct 30 05:52:33.447112 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447080 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:52:33.447240 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447219 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:52:33.447331 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447254 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:33.447420 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447401 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:52:33.447486 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447459 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:52:33.447548 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447488 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:52:33.447548 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447528 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:52:33.447548 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447531 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:52:33.447706 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447563 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:33.447706 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447595 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:52:33.447811 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447716 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:52:33.447811 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447731 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:52:33.447811 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447741 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:52:33.447811 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447774 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:52:33.447811 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.447787 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:52:33.448086 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.448056 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:52:33.456674 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.456629 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:52:33.456785 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.456685 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:52:33.462214 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462193 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-plsgv\"" Oct 30 05:52:33.462214 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462205 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d9dv5\"" Oct 30 05:52:33.462421 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462204 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:52:33.462421 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462285 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:52:33.462421 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462292 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vzk7w\"" Oct 30 05:52:33.462421 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462353 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q46lp\"" Oct 30 05:52:33.462421 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462358 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5hkvn\"" Oct 30 05:52:33.462421 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462409 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:52:33.462421 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462418 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q2ncw\"" Oct 30 05:52:33.462702 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462480 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:52:33.462702 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.462557 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mkzdc\"" Oct 30 05:52:33.512095 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-multus\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.512245 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xcr\" (UniqueName: \"kubernetes.io/projected/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-kube-api-access-44xcr\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.512245 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512128 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-env-overrides\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.512245 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-bin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.512245 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-etc-kubernetes\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.512245 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-modprobe-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.512245 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-run\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.512245 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512254 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-host\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-config\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-script-lib\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-hostroot\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-multus-certs\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-systemd\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqlk\" (UniqueName: \"kubernetes.io/projected/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-kube-api-access-zzqlk\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-bin\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-netd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512491 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-modprobe-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512496 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-hosts-file\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512535 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-hosts-file\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-tmp-dir\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-system-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysconfig\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.512642 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-device-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512622 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-systemd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-ovn\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovn-node-metrics-cert\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cni-binary-copy\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-k8s-cni-cncf-io\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512752 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-netns\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q88h\" (UniqueName: \"kubernetes.io/projected/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-kube-api-access-7q88h\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512785 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-sys-fs\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-netns\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a98d05d0-c37f-4a97-b943-fd4b728e29d0-host\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cnibin\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512855 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkcs\" (UniqueName: \"kubernetes.io/projected/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-kube-api-access-2fkcs\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-socket-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-log-socket\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512956 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-bin\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.513568 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.512971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-conf-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513010 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xcf5\" (UniqueName: \"kubernetes.io/projected/d2edb037-173f-4c27-987a-72d43df7fafb-kube-api-access-6xcf5\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-daemon-config\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513102 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-multus\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-agent-certs\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513171 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-sys\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsk8\" (UniqueName: \"kubernetes.io/projected/a98d05d0-c37f-4a97-b943-fd4b728e29d0-kube-api-access-bcsk8\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-conf\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-systemd-units\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-slash\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-var-lib-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-etc-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514116 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-tuned\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-system-cni-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-kubelet-config\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513489 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-socket-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513520 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-config\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513547 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513557 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-script-lib\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-netd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513632 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-lib-modules\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513728 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-run\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513760 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-env-overrides\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513782 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.514837 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513804 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-bin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-tmp\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513830 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-etc-kubernetes\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513905 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513996 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-hostroot\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514025 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-tmp-dir\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514034 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-multus-certs\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514075 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-system-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514077 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-systemd\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysconfig\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514117 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-log-socket\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514142 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-device-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514132 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrk9g\" (UniqueName: \"kubernetes.io/projected/b5705935-3763-404f-a84c-abc0133161bd-kube-api-access-vrk9g\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-socket-dir-parent\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-dbus\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514207 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-host\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514220 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:33.515605 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514237 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-node-log\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-conf-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a98d05d0-c37f-4a97-b943-fd4b728e29d0-serviceca\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsr4\" (UniqueName: \"kubernetes.io/projected/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-kube-api-access-dhsr4\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514296 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-systemd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5705935-3763-404f-a84c-abc0133161bd-iptables-alerter-script\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514350 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-kubernetes\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-var-lib-kubelet\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9876\" (UniqueName: \"kubernetes.io/projected/bd1a2002-0045-4025-8c34-23ab182ca2a6-kube-api-access-l9876\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-kubelet\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-konnectivity-ca\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-registration-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.514508 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-kubelet\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514566 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-kubelet\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514566 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-systemd-units\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514194 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.516574 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.513064 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-ovn\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.514601 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:34.014565713 +0000 UTC m=+3.146357624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.514643 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.514695 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret podName:9543ca89-5c01-4c07-9992-534b4955e331 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:34.014681483 +0000 UTC m=+3.146473381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret") pod "global-pull-secret-syncer-lbqgc" (UID: "9543ca89-5c01-4c07-9992-534b4955e331") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514726 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-node-log\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514749 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-slash\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-var-lib-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514836 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-etc-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.515122 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a98d05d0-c37f-4a97-b943-fd4b728e29d0-serviceca\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.515146 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-daemon-config\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.515239 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.514266 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-socket-dir-parent\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.516048 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5705935-3763-404f-a84c-abc0133161bd-iptables-alerter-script\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.516128 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-kubernetes\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.516188 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-var-lib-kubelet\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.516459 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-kubelet\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.517547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.516988 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-konnectivity-ca\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:33.518319 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.517067 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-registration-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.518319 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.517128 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-dbus\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:33.518319 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.517188 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-sys-fs\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.518319 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.517890 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cni-binary-copy\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.518319 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.517930 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-tuned\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.518319 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.518139 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-agent-certs\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:33.518319 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.518138 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-tmp\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.518319 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.518180 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovn-node-metrics-cert\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.523438 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523382 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-netns\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.523620 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5705935-3763-404f-a84c-abc0133161bd-host-slash\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.523681 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cnibin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.523754 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523703 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-sys\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.523754 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523724 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5705935-3763-404f-a84c-abc0133161bd-host-slash\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.523754 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523735 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cnibin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.524023 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523868 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.524023 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-conf\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.524023 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523929 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-netns\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.524023 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.523978 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-k8s-cni-cncf-io\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.524269 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.524030 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cnibin\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.524269 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.524090 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a98d05d0-c37f-4a97-b943-fd4b728e29d0-host\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.524269 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.524225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-system-cni-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.524551 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.524524 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.524625 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.524608 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-kubelet-config\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:33.524735 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.524715 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-lib-modules\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.525518 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.525490 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.525702 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.525503 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.537204 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.537072 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:33.537204 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.537095 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:33.537204 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.537110 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nzsf2 for pod openshift-network-diagnostics/network-check-target-2jjjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:33.537204 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:33.537176 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2 podName:dee312cb-ced0-47eb-8050-58bc39e462f6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:34.03715424 +0000 UTC m=+3.168946146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nzsf2" (UniqueName: "kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2") pod "network-check-target-2jjjp" (UID: "dee312cb-ced0-47eb-8050-58bc39e462f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:33.539673 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.539651 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xcr\" (UniqueName: \"kubernetes.io/projected/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-kube-api-access-44xcr\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.544266 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.544225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xcf5\" (UniqueName: \"kubernetes.io/projected/d2edb037-173f-4c27-987a-72d43df7fafb-kube-api-access-6xcf5\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.544592 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.544567 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrk9g\" (UniqueName: \"kubernetes.io/projected/b5705935-3763-404f-a84c-abc0133161bd-kube-api-access-vrk9g\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.544803 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.544778 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsr4\" (UniqueName: \"kubernetes.io/projected/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-kube-api-access-dhsr4\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.547543 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.547484 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9876\" (UniqueName: \"kubernetes.io/projected/bd1a2002-0045-4025-8c34-23ab182ca2a6-kube-api-access-l9876\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.551135 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.551114 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsk8\" (UniqueName: \"kubernetes.io/projected/a98d05d0-c37f-4a97-b943-fd4b728e29d0-kube-api-access-bcsk8\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.551230 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.551169 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqlk\" (UniqueName: \"kubernetes.io/projected/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-kube-api-access-zzqlk\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:33.551796 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.551777 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkcs\" (UniqueName: \"kubernetes.io/projected/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-kube-api-access-2fkcs\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.553095 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.553072 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q88h\" (UniqueName: \"kubernetes.io/projected/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-kube-api-access-7q88h\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.633987 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.633776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:52:33.642783 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.642759 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b4v22" Oct 30 05:52:33.652976 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.652952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2fn7x" Oct 30 05:52:33.653589 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:33.653558 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099ea3b3_43c1_4bbf_81aa_b3cbd803df66.slice/crio-1130b8d0ccbe242decea2dde9a37c1420b2a81304c99373cb746daf6d20774a4 WatchSource:0}: Error finding container 1130b8d0ccbe242decea2dde9a37c1420b2a81304c99373cb746daf6d20774a4: Status 404 returned error can't find the container with id 1130b8d0ccbe242decea2dde9a37c1420b2a81304c99373cb746daf6d20774a4 Oct 30 05:52:33.661751 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.661558 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:52:33.663135 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:33.663110 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87c9e48_d271_4c0c_92d3_38fbdac9ab2d.slice/crio-1d35b8efe205decd9dfa30e4ee3cf8d650ef51fedd0770cc1bc4801644abb193 WatchSource:0}: Error finding container 1d35b8efe205decd9dfa30e4ee3cf8d650ef51fedd0770cc1bc4801644abb193: Status 404 returned error can't find the container with id 1d35b8efe205decd9dfa30e4ee3cf8d650ef51fedd0770cc1bc4801644abb193 Oct 30 05:52:33.664300 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.664277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:52:33.671954 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.671754 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:33.673724 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:33.673692 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2edb037_173f_4c27_987a_72d43df7fafb.slice/crio-e1e19de3c90fb4f9fce255a3e1c8788b0cc04eeccbe08ea7d3b24057e22e2799 WatchSource:0}: Error finding container e1e19de3c90fb4f9fce255a3e1c8788b0cc04eeccbe08ea7d3b24057e22e2799: Status 404 returned error can't find the container with id e1e19de3c90fb4f9fce255a3e1c8788b0cc04eeccbe08ea7d3b24057e22e2799 Oct 30 05:52:33.688187 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:33.688158 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda975ec4d_2d0d_4c74_ba0b_56a9dee81442.slice/crio-210e30a3bb57bb25d3530b98b56362edec0d72cc3cada4f0104c292ca19aefb4 WatchSource:0}: Error finding container 210e30a3bb57bb25d3530b98b56362edec0d72cc3cada4f0104c292ca19aefb4: Status 404 returned error can't find the container with id 210e30a3bb57bb25d3530b98b56362edec0d72cc3cada4f0104c292ca19aefb4 Oct 30 05:52:33.700148 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.700131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:52:33.702377 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.702347 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:52:33.712266 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:33.712239 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98d05d0_c37f_4a97_b943_fd4b728e29d0.slice/crio-981bde37bab66f1ab27f4830eb9b63bde7078e6ff1d2cc10f06ff03c02380c8f WatchSource:0}: Error finding container 981bde37bab66f1ab27f4830eb9b63bde7078e6ff1d2cc10f06ff03c02380c8f: Status 404 returned error can't find the container with id 981bde37bab66f1ab27f4830eb9b63bde7078e6ff1d2cc10f06ff03c02380c8f Oct 30 05:52:33.715828 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.715807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:33.716224 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:33.716198 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bfdc3b_a4a7_4fcc_aa43_90ade5eda8d8.slice/crio-5d5757a57b39ee79973ecca2f9b79bf2998c1ada9b75cf9d3b2770a059a8fb5c WatchSource:0}: Error finding container 5d5757a57b39ee79973ecca2f9b79bf2998c1ada9b75cf9d3b2770a059a8fb5c: Status 404 returned error can't find the container with id 5d5757a57b39ee79973ecca2f9b79bf2998c1ada9b75cf9d3b2770a059a8fb5c Oct 30 05:52:33.720038 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:33.720020 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:33.726685 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:52:33.726659 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1a2002_0045_4025_8c34_23ab182ca2a6.slice/crio-507dc60e78c61cf1f15dab1770255d1e2c60543b1cdaaa92847a762be1f49a34 WatchSource:0}: Error finding container 507dc60e78c61cf1f15dab1770255d1e2c60543b1cdaaa92847a762be1f49a34: Status 404 returned error can't find the container with id 507dc60e78c61cf1f15dab1770255d1e2c60543b1cdaaa92847a762be1f49a34 Oct 30 05:52:34.028044 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.028015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:34.028214 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.028068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:34.028214 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.028171 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:34.028214 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.028185 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:34.028405 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.028239 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret podName:9543ca89-5c01-4c07-9992-534b4955e331 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:35.0282215 +0000 UTC m=+4.160013402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret") pod "global-pull-secret-syncer-lbqgc" (UID: "9543ca89-5c01-4c07-9992-534b4955e331") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:34.028405 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.028261 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:35.028250045 +0000 UTC m=+4.160041938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:34.129662 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.128856 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:34.129662 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.129155 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:34.129662 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.129180 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:34.129662 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.129194 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nzsf2 for pod openshift-network-diagnostics/network-check-target-2jjjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:34.129662 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.129254 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2 podName:dee312cb-ced0-47eb-8050-58bc39e462f6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:35.129235055 +0000 UTC m=+4.261026960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzsf2" (UniqueName: "kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2") pod "network-check-target-2jjjp" (UID: "dee312cb-ced0-47eb-8050-58bc39e462f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:34.365473 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.365049 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:47:33 +0000 UTC" deadline="2028-03-26 17:55:45.390976338 +0000 UTC" Oct 30 05:52:34.365473 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.365406 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12324h3m11.025580993s" Oct 30 05:52:34.436245 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.436215 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:34.436425 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.436372 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:34.436827 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.436806 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:34.436941 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.436919 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:34.437011 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.436996 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:34.437114 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:34.437096 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:34.455289 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.455257 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" event={"ID":"cc0cc40e9c1a761dcecd1aa2a7591d53","Type":"ContainerStarted","Data":"2ea61acd64e79a29e6c7c8bf2cd01054969e38199cbe4448480a4099c0906d71"} Oct 30 05:52:34.467049 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.467019 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" event={"ID":"bd1a2002-0045-4025-8c34-23ab182ca2a6","Type":"ContainerStarted","Data":"507dc60e78c61cf1f15dab1770255d1e2c60543b1cdaaa92847a762be1f49a34"} Oct 30 05:52:34.469741 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.469716 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v2swz" event={"ID":"a98d05d0-c37f-4a97-b943-fd4b728e29d0","Type":"ContainerStarted","Data":"981bde37bab66f1ab27f4830eb9b63bde7078e6ff1d2cc10f06ff03c02380c8f"} Oct 30 05:52:34.475052 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.475026 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnmml" event={"ID":"b5705935-3763-404f-a84c-abc0133161bd","Type":"ContainerStarted","Data":"0db0787edbd0cbce9e84487262579de70748640cfe4dc9bb968f8d23e0ce8807"} Oct 30 05:52:34.486582 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.486550 2573 generic.go:356] "Generic (PLEG): container finished" podID="7c8a05fdf5abb06be0c4a4fee3879794" containerID="e53d6128d59f87946bce79f37a156c87b364da02642e5c51dad0f5e3b86c67f0" exitCode=0 Oct 30 05:52:34.486866 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.486810 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" event={"ID":"7c8a05fdf5abb06be0c4a4fee3879794","Type":"ContainerDied","Data":"e53d6128d59f87946bce79f37a156c87b364da02642e5c51dad0f5e3b86c67f0"} Oct 30 05:52:34.493117 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.493093 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerStarted","Data":"5d5757a57b39ee79973ecca2f9b79bf2998c1ada9b75cf9d3b2770a059a8fb5c"} Oct 30 05:52:34.498696 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.498669 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kf896" event={"ID":"a975ec4d-2d0d-4c74-ba0b-56a9dee81442","Type":"ContainerStarted","Data":"210e30a3bb57bb25d3530b98b56362edec0d72cc3cada4f0104c292ca19aefb4"} Oct 30 05:52:34.504396 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.504358 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" event={"ID":"d2edb037-173f-4c27-987a-72d43df7fafb","Type":"ContainerStarted","Data":"e1e19de3c90fb4f9fce255a3e1c8788b0cc04eeccbe08ea7d3b24057e22e2799"} Oct 30 05:52:34.514701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.514675 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2fn7x" event={"ID":"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d","Type":"ContainerStarted","Data":"1d35b8efe205decd9dfa30e4ee3cf8d650ef51fedd0770cc1bc4801644abb193"} Oct 30 05:52:34.518873 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.518737 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b4v22" event={"ID":"099ea3b3-43c1-4bbf-81aa-b3cbd803df66","Type":"ContainerStarted","Data":"1130b8d0ccbe242decea2dde9a37c1420b2a81304c99373cb746daf6d20774a4"} Oct 30 05:52:34.522856 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.522805 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" event={"ID":"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c","Type":"ContainerStarted","Data":"7093c668e3dec801014005da7223089c7c2093e48b0e2f99508e93d9755192a8"} Oct 30 05:52:34.546318 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:34.545628 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" podStartSLOduration=1.545607313 podStartE2EDuration="1.545607313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:34.500188989 +0000 UTC m=+3.631980902" watchObservedRunningTime="2026-10-30 05:52:34.545607313 +0000 UTC m=+3.677399224" Oct 30 05:52:35.037196 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:35.037155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:35.037421 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:35.037231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:35.037421 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:35.037394 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:35.037541 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:35.037461 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret podName:9543ca89-5c01-4c07-9992-534b4955e331 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:37.037441153 +0000 UTC m=+6.169233055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret") pod "global-pull-secret-syncer-lbqgc" (UID: "9543ca89-5c01-4c07-9992-534b4955e331") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:35.037887 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:35.037870 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:35.037970 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:35.037924 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:37.037908187 +0000 UTC m=+6.169700088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:35.138079 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:35.138040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:35.138278 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:35.138259 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:35.138336 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:35.138291 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:35.138336 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:35.138305 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nzsf2 for pod openshift-network-diagnostics/network-check-target-2jjjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:35.138425 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:35.138378 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2 podName:dee312cb-ced0-47eb-8050-58bc39e462f6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:37.138359075 +0000 UTC m=+6.270150968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzsf2" (UniqueName: "kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2") pod "network-check-target-2jjjp" (UID: "dee312cb-ced0-47eb-8050-58bc39e462f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:35.366481 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:35.366350 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:47:33 +0000 UTC" deadline="2028-05-06 12:07:55.321508044 +0000 UTC" Oct 30 05:52:35.366481 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:35.366409 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13302h15m19.955103792s" Oct 30 05:52:35.532702 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:35.532617 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" event={"ID":"7c8a05fdf5abb06be0c4a4fee3879794","Type":"ContainerStarted","Data":"8ae5c9a6f8ae4b1ffc0a7421c488a7f4a4e0887e441306809850fdc0c0851276"} Oct 30 05:52:35.580916 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:35.580525 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" podStartSLOduration=2.580505744 podStartE2EDuration="2.580505744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:35.579765142 +0000 UTC m=+4.711557052" watchObservedRunningTime="2026-10-30 05:52:35.580505744 +0000 UTC m=+4.712297656" Oct 30 05:52:36.436589 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:36.436503 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:36.437050 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:36.436641 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:36.437166 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:36.437141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:36.437294 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:36.437260 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:36.437394 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:36.437324 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:36.437448 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:36.437414 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:37.053320 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:37.053279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:37.053523 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:37.053388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:37.053603 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:37.053528 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:37.053603 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:37.053595 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:41.053576563 +0000 UTC m=+10.185368455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:37.054013 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:37.053990 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:37.054118 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:37.054057 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret podName:9543ca89-5c01-4c07-9992-534b4955e331 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:41.054039169 +0000 UTC m=+10.185831061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret") pod "global-pull-secret-syncer-lbqgc" (UID: "9543ca89-5c01-4c07-9992-534b4955e331") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:37.153920 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:37.153877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:37.154105 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:37.154083 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:37.154189 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:37.154113 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:37.154189 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:37.154127 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nzsf2 for pod openshift-network-diagnostics/network-check-target-2jjjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:37.154291 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:37.154199 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2 podName:dee312cb-ced0-47eb-8050-58bc39e462f6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:41.154177305 +0000 UTC m=+10.285969210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzsf2" (UniqueName: "kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2") pod "network-check-target-2jjjp" (UID: "dee312cb-ced0-47eb-8050-58bc39e462f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:37.552350 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:37.552310 2573 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="a1133cb791e075085b2bedbe291447a6fff7cf54f6a201a34d7d40cbcd0e5281" exitCode=0 Oct 30 05:52:37.552902 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:37.552368 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerDied","Data":"a1133cb791e075085b2bedbe291447a6fff7cf54f6a201a34d7d40cbcd0e5281"} Oct 30 05:52:38.436711 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:38.436625 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:38.436711 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:38.436653 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:38.436943 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:38.436748 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:38.437253 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:38.437054 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:38.437253 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:38.437168 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:38.437253 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:38.437234 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:38.556794 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:38.556758 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kf896" event={"ID":"a975ec4d-2d0d-4c74-ba0b-56a9dee81442","Type":"ContainerStarted","Data":"8d30bed486c60dffe449b4e31e40b434102794d4a0fabbabe13ca2a1eda771c5"} Oct 30 05:52:39.563725 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:39.563637 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v2swz" event={"ID":"a98d05d0-c37f-4a97-b943-fd4b728e29d0","Type":"ContainerStarted","Data":"c03ec8eb38af8b1faea00be5d913da102db74660d5d365e318f1e2bbc00e39f6"} Oct 30 05:52:39.607060 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:39.607004 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v2swz" podStartSLOduration=2.489752883 podStartE2EDuration="7.6068537s" totalImagesPullingTime="5.117100817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:39.605922161 +0000 UTC m=+8.737714068" watchObservedRunningTime="2026-10-30 05:52:39.6068537 +0000 UTC m=+8.738645607" Oct 30 05:52:39.607185 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:39.607159 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kf896" podStartSLOduration=3.692242464 podStartE2EDuration="7.607150174s" totalImagesPullingTime="3.91490771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:38.604831364 +0000 UTC m=+7.736623271" watchObservedRunningTime="2026-10-30 05:52:39.607150174 +0000 UTC m=+8.738942082" Oct 30 05:52:40.436948 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:40.436114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:40.436948 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:40.436249 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:40.436948 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:40.436652 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:40.436948 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:40.436748 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:40.436948 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:40.436817 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:40.436948 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:40.436901 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:40.507742 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:40.507277 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:40.508391 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:40.508141 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:40.569098 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:40.569063 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" event={"ID":"d2edb037-173f-4c27-987a-72d43df7fafb","Type":"ContainerStarted","Data":"84f434497e0e3a5ad04404ef7edece8e0383d3c4c75de436aecf6fc661699613"} Oct 30 05:52:41.087095 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.087053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:41.087259 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.087137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:41.087341 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:41.087295 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:41.087393 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:41.087373 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:49.087354076 +0000 UTC m=+18.219145975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:41.087790 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:41.087739 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:41.087870 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:41.087794 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret podName:9543ca89-5c01-4c07-9992-534b4955e331 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:49.087777927 +0000 UTC m=+18.219569825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret") pod "global-pull-secret-syncer-lbqgc" (UID: "9543ca89-5c01-4c07-9992-534b4955e331") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:41.174920 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.174821 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:52:41.188328 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.188005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:41.188328 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:41.188161 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:41.188328 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:41.188178 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:41.188328 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:41.188187 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nzsf2 for pod openshift-network-diagnostics/network-check-target-2jjjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:41.188328 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:41.188233 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2 podName:dee312cb-ced0-47eb-8050-58bc39e462f6 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:49.188217016 +0000 UTC m=+18.320008914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzsf2" (UniqueName: "kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2") pod "network-check-target-2jjjp" (UID: "dee312cb-ced0-47eb-8050-58bc39e462f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:41.370054 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.369356 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:52:41.174836628Z","UUID":"2f1e0778-7d6d-4f3c-b812-ade0dba15b41","Handler":null,"Name":"","Endpoint":""} Oct 30 05:52:41.371722 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.371629 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:52:41.371722 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.371666 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:52:41.573516 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.573474 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" event={"ID":"d2edb037-173f-4c27-987a-72d43df7fafb","Type":"ContainerStarted","Data":"3a9bcbab651429183def1978f204bebb28d447b02b8aa51cf815beafc20c564c"} Oct 30 05:52:41.575478 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.575453 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:52:41.576527 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.576495 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2fn7x" event={"ID":"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d","Type":"ContainerStarted","Data":"16150d623ae1409db9e0474b5638f080e69d2ed701876ab19dc709767daf552d"} Oct 30 05:52:41.621833 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:41.621260 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2fn7x" podStartSLOduration=1.935383251 podStartE2EDuration="9.621240992s" totalImagesPullingTime="7.685857741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:41.620725892 +0000 UTC m=+10.752517795" watchObservedRunningTime="2026-10-30 05:52:41.621240992 +0000 UTC m=+10.753032902" Oct 30 05:52:42.435974 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:42.435772 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:42.436134 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:42.435786 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:42.436134 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:42.436076 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:42.436241 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:42.436182 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:42.436241 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:42.435833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:42.436413 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:42.436274 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:42.581539 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:42.581509 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" event={"ID":"d2edb037-173f-4c27-987a-72d43df7fafb","Type":"ContainerStarted","Data":"8073a8f482c05ceaeb90800421a9d2e952d65c8765934949ea517372e2b8b959"} Oct 30 05:52:42.583267 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:42.583241 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b4v22" event={"ID":"099ea3b3-43c1-4bbf-81aa-b3cbd803df66","Type":"ContainerStarted","Data":"eb382df381b2fd2288c6aaca45fc16c904bf5b92f516bf71a9815d9b72042a4e"} Oct 30 05:52:42.585390 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:42.585283 2573 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="b796259a12d9c90b68d6e9c74587fa68664f351ca6dc0e182093c2b269604c1d" exitCode=0 Oct 30 05:52:42.585390 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:42.585327 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerDied","Data":"b796259a12d9c90b68d6e9c74587fa68664f351ca6dc0e182093c2b269604c1d"} Oct 30 05:52:42.625748 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:42.625670 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" podStartSLOduration=2.538622597 podStartE2EDuration="10.625651106s" totalImagesPullingTime="8.087028509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:42.625529302 +0000 UTC m=+11.757321210" watchObservedRunningTime="2026-10-30 05:52:42.625651106 +0000 UTC m=+11.757443012" Oct 30 05:52:42.713244 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:42.713198 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-b4v22" podStartSLOduration=3.526708252 podStartE2EDuration="11.713170075s" totalImagesPullingTime="8.186461823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:42.71267731 +0000 UTC m=+11.844469237" watchObservedRunningTime="2026-10-30 05:52:42.713170075 +0000 UTC m=+11.844961985" Oct 30 05:52:43.590081 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:43.590041 2573 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="6738469b21b01c1799facc65579a403eda34caf530f6832c0923c9e69ba931b0" exitCode=0 Oct 30 05:52:43.590547 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:43.590127 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerDied","Data":"6738469b21b01c1799facc65579a403eda34caf530f6832c0923c9e69ba931b0"} Oct 30 05:52:43.592089 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:43.592061 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnmml" event={"ID":"b5705935-3763-404f-a84c-abc0133161bd","Type":"ContainerStarted","Data":"82ab2388b1f4414a5e2dd99a36bc6651bcb13f9ff1a48faaaefa693dbeaa6398"} Oct 30 05:52:43.720995 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:43.720957 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cnmml" podStartSLOduration=4.02874649 podStartE2EDuration="11.720944161s" totalImagesPullingTime="7.692197671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:43.720940656 +0000 UTC m=+12.852732561" watchObservedRunningTime="2026-10-30 05:52:43.720944161 +0000 UTC m=+12.852736066" Oct 30 05:52:44.436199 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:44.436157 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:44.436382 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:44.436207 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:44.436382 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:44.436298 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:44.436382 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:44.436294 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:44.436548 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:44.436398 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:44.436548 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:44.436494 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:44.595377 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:44.595341 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" event={"ID":"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c","Type":"ContainerStarted","Data":"8db70fba50d593c0afcf8032664509ea2a7abc15f85f16d97ed0b800a93a426c"} Oct 30 05:52:44.598511 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:44.598483 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerStarted","Data":"d7f19f90653557b5f7d19b1a9e493b64b9ffc148e788faf836889eda70689a71"} Oct 30 05:52:44.701239 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:44.701187 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" podStartSLOduration=3.737635873 podStartE2EDuration="13.70117496s" totalImagesPullingTime="9.963539087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:44.649160818 +0000 UTC m=+13.780952727" watchObservedRunningTime="2026-10-30 05:52:44.70117496 +0000 UTC m=+13.832966866" Oct 30 05:52:45.603681 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:45.603647 2573 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="d7f19f90653557b5f7d19b1a9e493b64b9ffc148e788faf836889eda70689a71" exitCode=0 Oct 30 05:52:45.604209 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:45.603725 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerDied","Data":"d7f19f90653557b5f7d19b1a9e493b64b9ffc148e788faf836889eda70689a71"} Oct 30 05:52:46.436752 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:46.436712 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:46.436917 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:46.436708 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:46.436917 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:46.436851 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:46.436917 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:46.436708 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:46.437060 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:46.436914 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:46.437060 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:46.436991 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:48.436064 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:48.435888 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:48.436462 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:48.435915 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:48.436462 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:48.436139 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:48.436462 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:48.435942 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:48.436462 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:48.436216 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:48.436462 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:48.436289 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:48.614777 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:48.614185 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerStarted","Data":"0dc0854da946bedf9fcf0a77144079108d707575b8e8e8ad9abfa8ef51e1f951"} Oct 30 05:52:48.615852 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:48.615826 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" event={"ID":"bd1a2002-0045-4025-8c34-23ab182ca2a6","Type":"ContainerStarted","Data":"b8965e758e90657150b7fad9beec4cb395701a6be2a8db23fc48ca5917523bf6"} Oct 30 05:52:49.153912 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.153704 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:49.153912 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.153889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:49.154074 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:49.153842 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:49.154074 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:49.153962 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:49.154074 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:49.153985 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:05.153970786 +0000 UTC m=+34.285762673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:49.154074 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:49.153998 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret podName:9543ca89-5c01-4c07-9992-534b4955e331 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:05.153992423 +0000 UTC m=+34.285784307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret") pod "global-pull-secret-syncer-lbqgc" (UID: "9543ca89-5c01-4c07-9992-534b4955e331") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:49.255020 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.254978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:49.255177 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:49.255123 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:49.255177 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:49.255138 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:49.255177 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:49.255147 2573 projected.go:196] Error preparing data for projected volume kube-api-access-nzsf2 for pod openshift-network-diagnostics/network-check-target-2jjjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:49.255279 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:49.255190 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2 podName:dee312cb-ced0-47eb-8050-58bc39e462f6 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:05.255179052 +0000 UTC m=+34.386970941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzsf2" (UniqueName: "kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2") pod "network-check-target-2jjjp" (UID: "dee312cb-ced0-47eb-8050-58bc39e462f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:49.620451 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.620424 2573 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="0dc0854da946bedf9fcf0a77144079108d707575b8e8e8ad9abfa8ef51e1f951" exitCode=0 Oct 30 05:52:49.620816 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.620512 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerDied","Data":"0dc0854da946bedf9fcf0a77144079108d707575b8e8e8ad9abfa8ef51e1f951"} Oct 30 05:52:49.623768 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.623748 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" event={"ID":"bd1a2002-0045-4025-8c34-23ab182ca2a6","Type":"ContainerStarted","Data":"b53fe8563f2927a3a065c8dfc0a1d8e96e696725977e266c0726585558a3c0ba"} Oct 30 05:52:49.623832 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.623807 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" event={"ID":"bd1a2002-0045-4025-8c34-23ab182ca2a6","Type":"ContainerStarted","Data":"47a4924eb10d46336a1cbaebce2641a30e5dcaaf930c7db16eb7e832a28535d3"} Oct 30 05:52:49.623862 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.623837 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" event={"ID":"bd1a2002-0045-4025-8c34-23ab182ca2a6","Type":"ContainerStarted","Data":"c730fbc7aa65a43a70964517f76d98792a7a76e38f125c85979c029167ef0f01"} Oct 30 05:52:49.623904 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.623869 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" event={"ID":"bd1a2002-0045-4025-8c34-23ab182ca2a6","Type":"ContainerStarted","Data":"1095208d93b64018a62fc746e9005766fcf63af30e6178f365fea5454cdee624"} Oct 30 05:52:49.623904 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.623897 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" event={"ID":"bd1a2002-0045-4025-8c34-23ab182ca2a6","Type":"ContainerStarted","Data":"90307a2727ef76e0e8563a5d2381b917a1b7ec32636d34403257d03c9f137067"} Oct 30 05:52:49.915989 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.915934 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:49.916124 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.916045 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:52:49.916447 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:49.916431 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kf896" Oct 30 05:52:50.436207 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:50.436187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:50.436380 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:50.436331 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:50.436380 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:50.436333 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:50.436452 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:50.436424 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:50.436496 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:50.436483 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:50.436585 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:50.436570 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:50.628852 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:50.628816 2573 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="bb7c91c48170e1dded3d7980f47e922e9a0f4b41d3c28e7d13b901ab09ed20e6" exitCode=0 Oct 30 05:52:50.629263 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:50.628864 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerDied","Data":"bb7c91c48170e1dded3d7980f47e922e9a0f4b41d3c28e7d13b901ab09ed20e6"} Oct 30 05:52:51.634375 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:51.634338 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" event={"ID":"bd1a2002-0045-4025-8c34-23ab182ca2a6","Type":"ContainerStarted","Data":"5ed2d0ad7822d3c046f50bedae8bd5e8a2dba3bbf15dfeb6eb240082a0a95841"} Oct 30 05:52:51.637443 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:51.637415 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82lcq" event={"ID":"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8","Type":"ContainerStarted","Data":"35edb1204f92901e96b8264c06414af3b0b15a32a74a6bfb0e731717ed3836d9"} Oct 30 05:52:51.712215 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:51.712178 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-82lcq" podStartSLOduration=8.521665651 podStartE2EDuration="19.71216705s" totalImagesPullingTime="11.190501399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:51.698626356 +0000 UTC m=+20.830418261" watchObservedRunningTime="2026-10-30 05:52:51.71216705 +0000 UTC m=+20.843958955" Oct 30 05:52:52.436489 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:52.436462 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:52.436652 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:52.436497 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:52.436652 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:52.436549 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:52.436652 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:52.436610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:52.436838 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:52.436721 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:52.436838 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:52.436810 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:53.645453 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:53.645418 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" event={"ID":"bd1a2002-0045-4025-8c34-23ab182ca2a6","Type":"ContainerStarted","Data":"d728b7744319e2e732dcaa81ed451c63fc9f8a533649ca8ca5c61a28525ccfd3"} Oct 30 05:52:53.646080 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:53.645660 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:53.661058 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:53.661034 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:53.740262 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:53.740206 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" podStartSLOduration=6.952241835 podStartE2EDuration="21.740193689s" totalImagesPullingTime="14.787951854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:53.739479063 +0000 UTC m=+22.871270973" watchObservedRunningTime="2026-10-30 05:52:53.740193689 +0000 UTC m=+22.871985594" Oct 30 05:52:54.435776 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:54.435745 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:54.435776 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:54.435765 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:54.435969 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:54.435749 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:54.435969 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:54.435841 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:54.435969 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:54.435910 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:54.436072 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:54.435975 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:54.647670 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:54.647644 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:52:54.648117 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:54.648095 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:54.662455 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:54.662434 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:52:55.037570 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:55.037499 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lbqgc"] Oct 30 05:52:55.037814 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:55.037616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:55.037814 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:55.037731 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:55.039701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:55.039675 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2jjjp"] Oct 30 05:52:55.039826 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:55.039766 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:55.039882 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:55.039853 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:55.047207 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:55.047188 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8ntvq"] Oct 30 05:52:55.047286 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:55.047274 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:55.047382 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:55.047367 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:55.649947 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:55.649921 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:52:56.435729 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:56.435686 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:56.435898 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:56.435769 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:56.435898 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:56.435855 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:56.435898 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:56.435867 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:56.435984 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:56.435934 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:56.436024 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:56.435990 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:52:56.652072 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:56.652048 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:52:58.436278 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:58.436251 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:52:58.436278 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:58.436262 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:52:58.436278 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:52:58.436272 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:52:58.436840 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:58.436376 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:52:58.436840 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:58.436422 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:52:58.436840 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:52:58.436486 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:53:00.436042 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:00.436010 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:00.436042 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:00.436037 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:00.436601 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:00.436122 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2jjjp" podUID="dee312cb-ced0-47eb-8050-58bc39e462f6" Oct 30 05:53:00.436601 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:00.436155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:00.436601 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:00.436228 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8ntvq" podUID="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" Oct 30 05:53:00.436601 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:00.436277 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbqgc" podUID="9543ca89-5c01-4c07-9992-534b4955e331" Oct 30 05:53:01.227530 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:01.227505 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:01.227764 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:01.227748 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:53:01.239126 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:01.239102 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:02.169613 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.169541 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeReady" Oct 30 05:53:02.169977 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.169664 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 05:53:02.239073 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.239040 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7b2h7"] Oct 30 05:53:02.284597 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.284566 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rmk2m"] Oct 30 05:53:02.284749 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.284683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.299812 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.299788 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7b2h7"] Oct 30 05:53:02.299935 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.299828 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jczv9"] Oct 30 05:53:02.299935 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.299922 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.304811 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.304786 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:53:02.304938 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.304921 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6cnz\"" Oct 30 05:53:02.309601 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.309584 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:53:02.313177 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.313161 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rmk2m"] Oct 30 05:53:02.313234 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.313198 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jczv9"] Oct 30 05:53:02.313290 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.313275 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:02.331487 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.331463 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:53:02.331590 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.331516 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:53:02.331590 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.331577 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:53:02.376531 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.376501 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-26ms6"] Oct 30 05:53:02.397027 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.397001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.402990 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.402933 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-26ms6"] Oct 30 05:53:02.404287 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.404266 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:53:02.404442 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.404414 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:53:02.404507 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.404492 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:53:02.404708 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.404691 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntdkp\"" Oct 30 05:53:02.404778 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.404709 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x8h66\"" Oct 30 05:53:02.404778 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.404773 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:53:02.436052 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.436001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:02.436127 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.436061 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:02.436127 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.436080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:02.449677 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.449656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k2qj\" (UniqueName: \"kubernetes.io/projected/d6557aa0-6639-40cd-baed-ea6bf194348c-kube-api-access-5k2qj\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:02.449770 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.449694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c8f0b4-b6a7-4055-abca-b49923505435-config-volume\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.449770 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.449720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.449862 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.449791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.449862 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.449839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.449938 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.449921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15d1e924-1f5b-4867-aa83-cd43814eebbb-crio-socket\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.449976 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.449956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15d1e924-1f5b-4867-aa83-cd43814eebbb-data-volume\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.450014 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.449999 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c8f0b4-b6a7-4055-abca-b49923505435-tmp-dir\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.450090 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.450068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrlz4\" (UniqueName: \"kubernetes.io/projected/f2c8f0b4-b6a7-4055-abca-b49923505435-kube-api-access-vrlz4\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.450143 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.450115 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6557aa0-6639-40cd-baed-ea6bf194348c-cert\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:02.450224 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.450210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gswk6\" (UniqueName: \"kubernetes.io/projected/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-api-access-gswk6\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.506757 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.506731 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 05:53:02.506866 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.506768 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 05:53:02.506866 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.506784 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:53:02.506866 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.506807 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:53:02.506866 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.506826 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 05:53:02.507833 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.507806 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:53:02.507833 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.507813 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xln8v\"" Oct 30 05:53:02.507978 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.507814 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-99gsm\"" Oct 30 05:53:02.507978 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.507842 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:53:02.507978 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.507862 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l4lh5\"" Oct 30 05:53:02.507978 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.507865 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 05:53:02.508143 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.508132 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nrzkq\"" Oct 30 05:53:02.551055 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.551161 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15d1e924-1f5b-4867-aa83-cd43814eebbb-crio-socket\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.551161 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15d1e924-1f5b-4867-aa83-cd43814eebbb-data-volume\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.551161 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551092 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-key\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.551161 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551115 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-cabundle\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.551389 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:02.551212 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:02.551389 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551227 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15d1e924-1f5b-4867-aa83-cd43814eebbb-crio-socket\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.551389 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c8f0b4-b6a7-4055-abca-b49923505435-tmp-dir\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.551389 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:02.551252 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls podName:f2c8f0b4-b6a7-4055-abca-b49923505435 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:03.051239821 +0000 UTC m=+32.183031705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls") pod "dns-default-7b2h7" (UID: "f2c8f0b4-b6a7-4055-abca-b49923505435") : secret "dns-default-metrics-tls" not found Oct 30 05:53:02.551389 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrlz4\" (UniqueName: \"kubernetes.io/projected/f2c8f0b4-b6a7-4055-abca-b49923505435-kube-api-access-vrlz4\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.551389 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6557aa0-6639-40cd-baed-ea6bf194348c-cert\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:02.551701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551417 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gswk6\" (UniqueName: \"kubernetes.io/projected/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-api-access-gswk6\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.551701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551454 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg582\" (UniqueName: \"kubernetes.io/projected/e2f5e019-7215-4c3d-8144-a32a9bba9965-kube-api-access-xg582\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.551701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551475 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15d1e924-1f5b-4867-aa83-cd43814eebbb-data-volume\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.551701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551486 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2qj\" (UniqueName: \"kubernetes.io/projected/d6557aa0-6639-40cd-baed-ea6bf194348c-kube-api-access-5k2qj\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:02.551701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551611 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c8f0b4-b6a7-4055-abca-b49923505435-config-volume\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.551701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551637 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c8f0b4-b6a7-4055-abca-b49923505435-tmp-dir\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.551701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.551701 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.551673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.552136 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:02.551745 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:53:02.552136 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:02.551747 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:02.552136 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:02.551804 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:03.051788109 +0000 UTC m=+32.183579992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : configmap "kube-rbac-proxy" not found Oct 30 05:53:02.552136 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:02.551824 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:03.051814738 +0000 UTC m=+32.183606628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:02.552333 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.552199 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c8f0b4-b6a7-4055-abca-b49923505435-config-volume\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.554904 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.554887 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6557aa0-6639-40cd-baed-ea6bf194348c-cert\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:02.579180 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.579159 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrlz4\" (UniqueName: \"kubernetes.io/projected/f2c8f0b4-b6a7-4055-abca-b49923505435-kube-api-access-vrlz4\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:02.581554 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.581534 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2qj\" (UniqueName: \"kubernetes.io/projected/d6557aa0-6639-40cd-baed-ea6bf194348c-kube-api-access-5k2qj\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:02.581635 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.581560 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gswk6\" (UniqueName: \"kubernetes.io/projected/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-api-access-gswk6\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:02.651938 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.651918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-key\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.652027 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.651946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-cabundle\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.652185 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.652167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg582\" (UniqueName: \"kubernetes.io/projected/e2f5e019-7215-4c3d-8144-a32a9bba9965-kube-api-access-xg582\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.652631 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.652612 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-cabundle\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.653992 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.653972 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-key\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.676748 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.676730 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg582\" (UniqueName: \"kubernetes.io/projected/e2f5e019-7215-4c3d-8144-a32a9bba9965-kube-api-access-xg582\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:02.697070 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.697052 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:02.714218 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:02.714197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:03.058278 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:03.058212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:03.058278 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:03.058251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:03.058494 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:03.058281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:03.058494 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:03.058372 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:53:03.058494 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:03.058408 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:03.058494 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:03.058379 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:03.058494 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:03.058450 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:04.058435904 +0000 UTC m=+33.190227819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : configmap "kube-rbac-proxy" not found Oct 30 05:53:03.058494 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:03.058465 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls podName:f2c8f0b4-b6a7-4055-abca-b49923505435 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:04.058459122 +0000 UTC m=+33.190251006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls") pod "dns-default-7b2h7" (UID: "f2c8f0b4-b6a7-4055-abca-b49923505435") : secret "dns-default-metrics-tls" not found Oct 30 05:53:03.058494 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:03.058476 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:04.058470497 +0000 UTC m=+33.190262380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:03.061631 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:03.061614 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jczv9"] Oct 30 05:53:03.063257 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:53:03.063237 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6557aa0_6639_40cd_baed_ea6bf194348c.slice/crio-af2424601a36112818c4c1d61c7fa61a5cfb6671539dbca51cde3b0552cecf13 WatchSource:0}: Error finding container af2424601a36112818c4c1d61c7fa61a5cfb6671539dbca51cde3b0552cecf13: Status 404 returned error can't find the container with id af2424601a36112818c4c1d61c7fa61a5cfb6671539dbca51cde3b0552cecf13 Oct 30 05:53:03.074467 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:03.074445 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-26ms6"] Oct 30 05:53:03.075216 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:53:03.075193 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f5e019_7215_4c3d_8144_a32a9bba9965.slice/crio-5397819fe4d9189ccd1df5648f17d45425d8136716feabf5b4186dc04b18842e WatchSource:0}: Error finding container 5397819fe4d9189ccd1df5648f17d45425d8136716feabf5b4186dc04b18842e: Status 404 returned error can't find the container with id 5397819fe4d9189ccd1df5648f17d45425d8136716feabf5b4186dc04b18842e Oct 30 05:53:03.668113 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:03.668074 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jczv9" event={"ID":"d6557aa0-6639-40cd-baed-ea6bf194348c","Type":"ContainerStarted","Data":"af2424601a36112818c4c1d61c7fa61a5cfb6671539dbca51cde3b0552cecf13"} Oct 30 05:53:03.669350 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:03.669280 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" event={"ID":"e2f5e019-7215-4c3d-8144-a32a9bba9965","Type":"ContainerStarted","Data":"5397819fe4d9189ccd1df5648f17d45425d8136716feabf5b4186dc04b18842e"} Oct 30 05:53:04.065043 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:04.065006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:04.065221 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:04.065053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:04.065221 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:04.065134 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:53:04.065221 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:04.065158 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:04.065221 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:04.065193 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:06.065177061 +0000 UTC m=+35.196968959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : configmap "kube-rbac-proxy" not found Oct 30 05:53:04.065221 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:04.065222 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:06.065205114 +0000 UTC m=+35.196997001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:04.065511 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:04.065250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:04.065511 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:04.065361 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:04.065511 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:04.065386 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls podName:f2c8f0b4-b6a7-4055-abca-b49923505435 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:06.065379541 +0000 UTC m=+35.197171424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls") pod "dns-default-7b2h7" (UID: "f2c8f0b4-b6a7-4055-abca-b49923505435") : secret "dns-default-metrics-tls" not found Oct 30 05:53:04.974577 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:04.974548 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b4v22_099ea3b3-43c1-4bbf-81aa-b3cbd803df66/dns-node-resolver/0.log" Oct 30 05:53:05.174842 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.174229 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:05.174842 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.174332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:05.174842 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:05.174604 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:05.174842 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:05.174673 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:37.174654715 +0000 UTC m=+66.306446600 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : secret "metrics-daemon-secret" not found Oct 30 05:53:05.177278 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.177250 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:05.213747 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.213718 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:05.275008 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.274975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:05.277776 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.277753 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:05.533573 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.533541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:05.541505 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.541483 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lbqgc"] Oct 30 05:53:05.543052 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:53:05.543030 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9543ca89_5c01_4c07_9992_534b4955e331.slice/crio-719219675a61628d482afec05c9300cc099f35136c0eb43dc3214eaa6a280f83 WatchSource:0}: Error finding container 719219675a61628d482afec05c9300cc099f35136c0eb43dc3214eaa6a280f83: Status 404 returned error can't find the container with id 719219675a61628d482afec05c9300cc099f35136c0eb43dc3214eaa6a280f83 Oct 30 05:53:05.674687 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.674649 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lbqgc" event={"ID":"9543ca89-5c01-4c07-9992-534b4955e331","Type":"ContainerStarted","Data":"719219675a61628d482afec05c9300cc099f35136c0eb43dc3214eaa6a280f83"} Oct 30 05:53:05.845813 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:05.845792 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2jjjp"] Oct 30 05:53:05.993554 ip-10-0-136-123 kubenswrapper[2573]: W1030 05:53:05.993528 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee312cb_ced0_47eb_8050_58bc39e462f6.slice/crio-a9645f0dc5f838c70bdac34995724a404c3095df1207e41fe6b92ce7980fd36f WatchSource:0}: Error finding container a9645f0dc5f838c70bdac34995724a404c3095df1207e41fe6b92ce7980fd36f: Status 404 returned error can't find the container with id a9645f0dc5f838c70bdac34995724a404c3095df1207e41fe6b92ce7980fd36f Oct 30 05:53:06.081751 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.081724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:06.081872 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.081756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:06.081872 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.081788 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:06.081872 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:06.081861 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:53:06.082015 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:06.081940 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.081918959 +0000 UTC m=+39.213710860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : configmap "kube-rbac-proxy" not found Oct 30 05:53:06.082015 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:06.081863 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:06.082015 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:06.081985 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.081973908 +0000 UTC m=+39.213765805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:06.082015 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:06.081868 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:06.082015 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:06.082012 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls podName:f2c8f0b4-b6a7-4055-abca-b49923505435 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:10.082003059 +0000 UTC m=+39.213794958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls") pod "dns-default-7b2h7" (UID: "f2c8f0b4-b6a7-4055-abca-b49923505435") : secret "dns-default-metrics-tls" not found Oct 30 05:53:06.175690 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.175612 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v2swz_a98d05d0-c37f-4a97-b943-fd4b728e29d0/node-ca/0.log" Oct 30 05:53:06.679088 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.679048 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2jjjp" event={"ID":"dee312cb-ced0-47eb-8050-58bc39e462f6","Type":"ContainerStarted","Data":"a9645f0dc5f838c70bdac34995724a404c3095df1207e41fe6b92ce7980fd36f"} Oct 30 05:53:06.681218 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.681188 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" event={"ID":"e2f5e019-7215-4c3d-8144-a32a9bba9965","Type":"ContainerStarted","Data":"41d4aa30b02a5ef111378d8f23e1abe7b1fc279ecf703f23af168fd14823f2aa"} Oct 30 05:53:06.684363 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.684338 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jczv9" event={"ID":"d6557aa0-6639-40cd-baed-ea6bf194348c","Type":"ContainerStarted","Data":"87861aeee654935348cacd6e6da64e29f9dcc62e89c2f824295993e4e499b569"} Oct 30 05:53:06.726702 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.726654 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" podStartSLOduration=1.786708817 podStartE2EDuration="4.726639556s" totalImagesPullingTime="2.939930739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:06.726115588 +0000 UTC m=+35.857907494" watchObservedRunningTime="2026-10-30 05:53:06.726639556 +0000 UTC m=+35.858431501" Oct 30 05:53:06.785472 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.785411 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jczv9" podStartSLOduration=1.823284328 podStartE2EDuration="4.785390394s" totalImagesPullingTime="2.962106066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:06.783511307 +0000 UTC m=+35.915303215" watchObservedRunningTime="2026-10-30 05:53:06.785390394 +0000 UTC m=+35.917182301" Oct 30 05:53:06.785659 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:06.785604 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jczv9_d6557aa0-6639-40cd-baed-ea6bf194348c/serve-healthcheck-canary/0.log" Oct 30 05:53:09.694634 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:09.694605 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2jjjp" event={"ID":"dee312cb-ced0-47eb-8050-58bc39e462f6","Type":"ContainerStarted","Data":"08e4f2553337a7c58f0d7ddf2f97d775a5107e12e9f514e7a7f3d0c03a61700b"} Oct 30 05:53:09.695081 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:09.694745 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:09.738086 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:09.738041 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2jjjp" podStartSLOduration=34.198724017 podStartE2EDuration="37.738026336s" totalImagesPullingTime="3.539302319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:09.737438868 +0000 UTC m=+38.869230774" watchObservedRunningTime="2026-10-30 05:53:09.738026336 +0000 UTC m=+38.869818242" Oct 30 05:53:10.125104 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:10.125071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:10.125104 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:10.125107 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:10.125371 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:10.125131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:10.125371 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:10.125251 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:10.125371 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:10.125324 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls podName:f2c8f0b4-b6a7-4055-abca-b49923505435 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:18.125295932 +0000 UTC m=+47.257087816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls") pod "dns-default-7b2h7" (UID: "f2c8f0b4-b6a7-4055-abca-b49923505435") : secret "dns-default-metrics-tls" not found Oct 30 05:53:10.125371 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:10.125249 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:10.125560 ip-10-0-136-123 kubenswrapper[2573]: E1030 05:53:10.125415 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:18.125396841 +0000 UTC m=+47.257188734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:10.125678 ip-10-0-136-123 kubenswrapper[2573]: I1030 05:53:10.125653 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:11.128189 ip-10-0-136-123 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 05:53:11.147693 ip-10-0-136-123 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 05:53:11.147851 ip-10-0-136-123 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 05:53:11.147897 ip-10-0-136-123 systemd[1]: kubelet.service: Consumed 2.175s CPU time. Oct 30 05:53:11.158046 ip-10-0-136-123 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:53:11.261487 ip-10-0-136-123 kubenswrapper[4431]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:53:11.261487 ip-10-0-136-123 kubenswrapper[4431]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:53:11.261487 ip-10-0-136-123 kubenswrapper[4431]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:53:11.261487 ip-10-0-136-123 kubenswrapper[4431]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:53:11.263541 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263467 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:11.263541 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263541 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263553 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263557 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263560 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263563 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263566 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263569 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263571 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263574 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263585 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263589 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263591 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263594 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263597 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263601 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263603 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263606 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263609 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263611 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263614 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:11.263616 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263617 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263620 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263623 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263627 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263632 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263635 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263638 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263641 4431 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263644 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263646 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263649 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263653 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263657 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263660 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263664 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263666 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263669 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263671 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263674 4431 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263677 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:11.264118 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263679 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263681 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263684 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263687 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263690 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263692 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263694 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263697 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263699 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263702 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263704 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263707 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263710 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263712 4431 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263715 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263718 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263721 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263723 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263726 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:11.264632 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263729 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263732 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263735 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263737 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263740 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263742 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263745 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263747 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263750 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263752 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263755 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263758 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263760 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263763 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263765 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263768 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263771 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263773 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263776 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:11.265107 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263780 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263783 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263786 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263788 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263791 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263793 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263796 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263800 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263803 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263806 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263808 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263811 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263814 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263816 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263819 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263821 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263824 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263827 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263830 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:11.265594 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263832 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263835 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263837 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263840 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263842 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263845 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263847 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.263850 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264591 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264599 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264602 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264613 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264616 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264619 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264622 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264624 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264627 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264629 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264632 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264635 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:11.266108 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264638 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264642 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264645 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264648 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264650 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264653 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264655 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264658 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264660 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264663 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264665 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264668 4431 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264670 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264673 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264676 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264679 4431 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264682 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264684 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264687 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264689 4431 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:11.266620 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264692 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264695 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264697 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264700 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264709 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264712 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264714 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264717 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264720 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264722 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264725 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264727 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264730 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264732 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264735 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264737 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264740 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264743 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264745 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:11.267155 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264748 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264750 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264753 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264756 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264759 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264762 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264765 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264769 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264773 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264776 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264779 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264781 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264784 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264786 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264789 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264792 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264794 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264799 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264807 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:11.267655 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264810 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264813 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264815 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264818 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264821 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264823 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264826 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264828 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264831 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264833 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264836 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264839 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264841 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264844 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264847 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264849 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264852 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264855 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264857 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:11.268173 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264860 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264862 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264866 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264870 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264874 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264877 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264879 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264882 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264885 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264888 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264891 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264893 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264896 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264899 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264901 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264904 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.264908 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265375 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265381 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265384 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:11.268660 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265387 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265390 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265393 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265396 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265402 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265404 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265408 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265410 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265414 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265416 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265420 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265422 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265425 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265428 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265431 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265434 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265437 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265439 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265442 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:11.269212 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265445 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265448 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265451 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265454 4431 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265457 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265459 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265463 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265467 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265470 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265473 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265476 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265479 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265481 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265484 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265487 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265490 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265492 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265501 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265506 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:11.269707 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265511 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265515 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265519 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265522 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265527 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265531 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265534 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265537 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265540 4431 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265542 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265545 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265552 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265554 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265557 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265560 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265563 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265565 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265568 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265570 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265573 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:11.270170 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265577 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265580 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265582 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265585 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265588 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265590 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265593 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265595 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265598 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265601 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265605 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265607 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265610 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265613 4431 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265615 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265618 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265620 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265623 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265625 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265628 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:11.270682 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265631 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265635 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265639 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265643 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265646 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265649 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265651 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265654 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265656 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265659 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265662 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265664 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265666 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265670 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265673 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265675 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265678 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265681 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265683 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:11.271179 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265685 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265688 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265690 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265693 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265696 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.265698 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265798 4431 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265816 4431 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265822 4431 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265826 4431 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265831 4431 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265834 4431 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265839 4431 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265843 4431 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265846 4431 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265849 4431 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265854 4431 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265857 4431 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265860 4431 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265863 4431 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265866 4431 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265870 4431 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265873 4431 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:53:11.271687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265876 4431 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265880 4431 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265883 4431 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265886 4431 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265890 4431 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265894 4431 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265898 4431 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265901 4431 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265904 4431 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265907 4431 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265910 4431 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265913 4431 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265917 4431 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265920 4431 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265923 4431 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265927 4431 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265930 4431 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265933 4431 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265936 4431 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265939 4431 flags.go:64] FLAG: --enable-server="true" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265942 4431 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265947 4431 flags.go:64] FLAG: --event-burst="100" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265950 4431 flags.go:64] FLAG: --event-qps="50" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265953 4431 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:53:11.272240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265956 4431 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265961 4431 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265965 4431 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265968 4431 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265971 4431 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265974 4431 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265977 4431 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265980 4431 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265983 4431 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265986 4431 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265989 4431 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265992 4431 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.265995 4431 flags.go:64] FLAG: --feature-gates="" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266002 4431 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266006 4431 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266009 4431 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266012 4431 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266015 4431 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266018 4431 flags.go:64] FLAG: --help="false" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266021 4431 flags.go:64] FLAG: --hostname-override="ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266024 4431 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266027 4431 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266031 4431 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266034 4431 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:53:11.272890 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266037 4431 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266040 4431 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266043 4431 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266046 4431 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266049 4431 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266052 4431 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266055 4431 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266059 4431 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266062 4431 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266066 4431 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266069 4431 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266072 4431 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266075 4431 flags.go:64] FLAG: --lock-file="" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266079 4431 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266082 4431 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266085 4431 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266090 4431 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266093 4431 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266096 4431 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266099 4431 flags.go:64] FLAG: --logging-format="text" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266102 4431 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266106 4431 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266109 4431 flags.go:64] FLAG: --manifest-url="" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266112 4431 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266116 4431 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:53:11.273480 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266119 4431 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266124 4431 flags.go:64] FLAG: --max-pods="110" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266127 4431 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266132 4431 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266135 4431 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266138 4431 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266141 4431 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266144 4431 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266147 4431 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266153 4431 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266156 4431 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266160 4431 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266163 4431 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266166 4431 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266169 4431 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266172 4431 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266176 4431 flags.go:64] FLAG: --port="10250" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266179 4431 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266182 4431 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b77122a4542ef9cd" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266185 4431 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266188 4431 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266191 4431 flags.go:64] FLAG: --register-node="true" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266194 4431 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266198 4431 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266201 4431 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266204 4431 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:53:11.274565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266207 4431 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266211 4431 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266214 4431 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266217 4431 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266220 4431 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266223 4431 flags.go:64] FLAG: --runonce="false" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266227 4431 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266230 4431 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266233 4431 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266237 4431 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266240 4431 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266243 4431 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266246 4431 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266249 4431 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266252 4431 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266254 4431 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266257 4431 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266260 4431 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266263 4431 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266266 4431 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266269 4431 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266274 4431 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266279 4431 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266282 4431 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266287 4431 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:53:11.275525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266289 4431 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266292 4431 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266295 4431 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266310 4431 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266314 4431 flags.go:64] FLAG: --v="2" Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266318 4431 flags.go:64] FLAG: --version="false" Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266322 4431 flags.go:64] FLAG: --vmodule="" Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266326 4431 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266329 4431 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266456 4431 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266459 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266462 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266466 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266469 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266472 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266475 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266479 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266482 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266484 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266487 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266490 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266492 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:11.276244 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266495 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266498 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266501 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266503 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266506 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266509 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266511 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266515 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266518 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266520 4431 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266523 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266526 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266528 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266531 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266534 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266537 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266539 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266542 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266544 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:11.277019 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266547 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266550 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266552 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266555 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266558 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266560 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266563 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266567 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266570 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266572 4431 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266575 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266577 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266580 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266583 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266585 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266588 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266591 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266594 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266596 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266600 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:11.277611 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266605 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266609 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266612 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266615 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266617 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266620 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266623 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266625 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266628 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266631 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266633 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266636 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266638 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266641 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266644 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266646 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266649 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266652 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266655 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:11.278275 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266658 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266661 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266665 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266669 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266672 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266675 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266678 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266680 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266683 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266685 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266688 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266691 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266693 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266697 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266700 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266703 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266705 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266708 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266711 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:11.278764 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266713 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266716 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266719 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266721 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266724 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266726 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266729 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266732 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266734 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266737 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266740 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266742 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266745 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266749 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266752 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266754 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:11.279413 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.266779 4431 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266812 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266816 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266819 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266822 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266825 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266828 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266831 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266833 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266836 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266839 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266843 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266845 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266848 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266851 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266854 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266856 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266859 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266862 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266864 4431 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:11.279899 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266867 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266869 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266872 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266875 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266878 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266880 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266883 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266885 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266888 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266892 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266895 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266898 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266900 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266903 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266906 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266908 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266911 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266913 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266916 4431 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266918 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:11.280623 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266921 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266924 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266926 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266930 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266933 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266935 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266938 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266941 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266945 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266948 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266951 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266954 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266957 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266959 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266962 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266965 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266967 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266970 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266972 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:11.281487 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266975 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266978 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266981 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266984 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266987 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266991 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266993 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266996 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.266999 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267001 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267004 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267008 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267011 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267014 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267017 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267019 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267024 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267026 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267029 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:11.282349 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267032 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267034 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267037 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267040 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267043 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267045 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267048 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267051 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267053 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267056 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267058 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267061 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267064 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267066 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267069 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267072 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267075 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267078 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267082 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:11.283121 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267084 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267088 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267090 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267109 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267112 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267115 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267118 4431 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267121 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267124 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.267127 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.270675 4431 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.270690 4431 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.270707 4431 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.270712 4431 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270766 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270771 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270774 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270777 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270780 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270783 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:11.284064 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270785 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270788 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270791 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270794 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270796 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270799 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270802 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270805 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270808 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270811 4431 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270815 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270818 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270822 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270826 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270829 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270833 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270837 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270841 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270844 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:11.284925 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270846 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270849 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270852 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270855 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270857 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270860 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270863 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270865 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270868 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270871 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270873 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270876 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270879 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270881 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270884 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270887 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270889 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270892 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270895 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270897 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:11.285887 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270912 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270916 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270919 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270923 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270926 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270929 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270932 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270934 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270937 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270940 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270942 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270945 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270948 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270951 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270953 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270956 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270958 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270961 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270964 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270966 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:11.286797 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270969 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270971 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270974 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270976 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270979 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270981 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270984 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270986 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270989 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270992 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270994 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270997 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.270999 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271002 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271005 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271008 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271010 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271014 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271016 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271019 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:11.287407 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271021 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271024 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271027 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271029 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271032 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271034 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271037 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271039 4431 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271042 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271044 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271047 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271049 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271052 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271054 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271057 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271060 4431 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271062 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271065 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271067 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:11.287940 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271070 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271073 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.271099 4431 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271229 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271234 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271238 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271241 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271244 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271247 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271250 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271252 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271255 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271258 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271261 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271263 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271266 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:11.288514 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271268 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271271 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271274 4431 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271277 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271279 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271282 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271285 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271287 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271290 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271292 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271295 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271313 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271318 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271322 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271326 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271330 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271333 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271335 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271338 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271341 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:11.288905 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271343 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271346 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271349 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271351 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271354 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271357 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271360 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271362 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271366 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271368 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271371 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271374 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271376 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271379 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271382 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271384 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271387 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271391 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271395 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:11.289417 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271398 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271401 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271404 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271406 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271409 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271411 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271414 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271416 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271419 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271422 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271424 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271426 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271429 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271432 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271434 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271437 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271440 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271442 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271445 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271447 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:11.289871 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271450 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271453 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271456 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271459 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271461 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271464 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271466 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271469 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271472 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271474 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271477 4431 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271479 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271482 4431 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271484 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271487 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271489 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271492 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271495 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271497 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:11.290370 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271500 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271503 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271507 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271511 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271514 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271516 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271519 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271521 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271524 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271527 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271531 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271534 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271536 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271539 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271542 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:11.290841 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.271571 4431 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271604 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271608 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271611 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271614 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271616 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271619 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271621 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271624 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271627 4431 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271629 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271632 4431 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271634 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271637 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271639 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271642 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271645 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271647 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271650 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:11.291238 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271654 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271658 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271662 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271664 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271667 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271669 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271672 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271674 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271683 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271686 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271689 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271692 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271694 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271697 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271700 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271703 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271706 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271708 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271711 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:11.291906 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271714 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271716 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271718 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271721 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271724 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271726 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271729 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271731 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271734 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271737 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271739 4431 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271742 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271744 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271747 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271750 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271752 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271755 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271757 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271760 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271762 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:11.292732 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271765 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271768 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271771 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271773 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271776 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271778 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271781 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271784 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271786 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271789 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271792 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271795 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271797 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271799 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271802 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271805 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271808 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271810 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271813 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:11.293508 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271815 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271818 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271821 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271824 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271826 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271829 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271833 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271837 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271840 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271843 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271846 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271849 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271851 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271854 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271856 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271859 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271862 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271864 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271867 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:11.293996 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271869 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271872 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271874 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271878 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271880 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271883 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271886 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271888 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271891 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271893 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:11.271896 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.271941 4431 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.272926 4431 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.272975 4431 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.273152 4431 server.go:1014] "Starting client certificate rotation" Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.273287 4431 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.274509 4431 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:53:11.294502 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.273513 4431 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:47:31 +0000 UTC" deadline="2028-04-12 03:21:10.202731017 +0000 UTC" Oct 30 05:53:11.294950 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.274887 4431 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12717h27m58.92785184s" Oct 30 05:53:11.294950 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.276042 4431 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:53:11.294950 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.277565 4431 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:53:11.294950 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.279178 4431 log.go:25] "Validated CRI v1 image API" Oct 30 05:53:11.294950 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.279981 4431 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:53:11.294950 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.282053 4431 fs.go:135] Filesystem UUIDs: map[59820a08-003b-4739-a183-165dd122251e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e7c99e0a-7a8b-4bea-8062-c868a1269592:/dev/nvme0n1p3] Oct 30 05:53:11.295124 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.282080 4431 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0db0787edbd0cbce9e84487262579de70748640cfe4dc9bb968f8d23e0ce8807/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0db0787edbd0cbce9e84487262579de70748640cfe4dc9bb968f8d23e0ce8807/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1130b8d0ccbe242decea2dde9a37c1420b2a81304c99373cb746daf6d20774a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1130b8d0ccbe242decea2dde9a37c1420b2a81304c99373cb746daf6d20774a4/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d35b8efe205decd9dfa30e4ee3cf8d650ef51fedd0770cc1bc4801644abb193/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d35b8efe205decd9dfa30e4ee3cf8d650ef51fedd0770cc1bc4801644abb193/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/210e30a3bb57bb25d3530b98b56362edec0d72cc3cada4f0104c292ca19aefb4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/210e30a3bb57bb25d3530b98b56362edec0d72cc3cada4f0104c292ca19aefb4/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/507dc60e78c61cf1f15dab1770255d1e2c60543b1cdaaa92847a762be1f49a34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/507dc60e78c61cf1f15dab1770255d1e2c60543b1cdaaa92847a762be1f49a34/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5397819fe4d9189ccd1df5648f17d45425d8136716feabf5b4186dc04b18842e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5397819fe4d9189ccd1df5648f17d45425d8136716feabf5b4186dc04b18842e/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d5757a57b39ee79973ecca2f9b79bf2998c1ada9b75cf9d3b2770a059a8fb5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d5757a57b39ee79973ecca2f9b79bf2998c1ada9b75cf9d3b2770a059a8fb5c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d68842197a388956e4be93267daa955a701e20d75801d34e8b4f24a7af19132/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d68842197a388956e4be93267daa955a701e20d75801d34e8b4f24a7af19132/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/719219675a61628d482afec05c9300cc099f35136c0eb43dc3214eaa6a280f83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/719219675a61628d482afec05c9300cc099f35136c0eb43dc3214eaa6a280f83/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/981bde37bab66f1ab27f4830eb9b63bde7078e6ff1d2cc10f06ff03c02380c8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/981bde37bab66f1ab27f4830eb9b63bde7078e6ff1d2cc10f06ff03c02380c8f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9645f0dc5f838c70bdac34995724a404c3095df1207e41fe6b92ce7980fd36f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9645f0dc5f838c70bdac34995724a404c3095df1207e41fe6b92ce7980fd36f/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af2424601a36112818c4c1d61c7fa61a5cfb6671539dbca51cde3b0552cecf13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af2424601a36112818c4c1d61c7fa61a5cfb6671539dbca51cde3b0552cecf13/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1e19de3c90fb4f9fce255a3e1c8788b0cc04eeccbe08ea7d3b24057e22e2799/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1e19de3c90fb4f9fce255a3e1c8788b0cc04eeccbe08ea7d3b24057e22e2799/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f549965acef59f07eb50a042f648b8e4c483d466fdb8322854e3e42ac59abfb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f549965acef59f07eb50a042f648b8e4c483d466fdb8322854e3e42ac59abfb7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/099ea3b3-43c1-4bbf-81aa-b3cbd803df66/volumes/kubernetes.io~projected/kube-api-access-dhsr4:{mountpoint:/var/lib/kubelet/pods/099ea3b3-43c1-4bbf-81aa-b3cbd803df66/volumes/kubernetes.io~projected/kube-api-access-dhsr4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8/volumes/kubernetes.io~projected/kube-api-access-7q88h:{mountpoint:/var/lib/kubelet/pods/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8/volumes/kubernetes.io~projected/kube-api-access-7q88h major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15d1e924-1f5b-4867-aa83-cd43814eebbb/volumes/kubernetes.io~projected/kube-api-access-gswk6:{mountpoint:/var/lib/kubelet/pods/15d1e924-1f5b-4867-aa83-cd43814eebbb/volumes/kubernetes.io~projected/kube-api-access-gswk6 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6/volumes/kubernetes.io~projected/kube-api-access-zzqlk:{mountpoint:/var/lib/kubelet/pods/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6/volumes/kubernetes.io~projected/kube-api-access-zzqlk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9543ca89-5c01-4c07-9992-534b4955e331/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9543ca89-5c01-4c07-9992-534b4955e331/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c/volumes/kubernetes.io~projected/kube-api-access-2fkcs:{mountpoint:/var/lib/kubelet/pods/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c/volumes/kubernetes.io~projected/kube-api-access-2fkcs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a975ec4d-2d0d-4c74-ba0b-56a9dee81442/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a975ec4d-2d0d-4c74-ba0b-56a9dee81442/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a98d05d0-c37f-4a97-b943-fd4b728e29d0/volumes/kubernetes.io~projected/kube-api-access-bcsk8:{mountpoint:/var/lib/kubelet/pods/a98d05d0-c37f-4a97-b943-fd4b728e29d0/volumes/kubernetes.io~projected/kube-api-access-bcsk8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5705935-3763-404f-a84c-abc0133161bd/volumes/kubernetes.io~projected/kube-api-access-vrk9g:{mountpoint:/var/lib/kubelet/pods/b5705935-3763-404f-a84c-abc0133161bd/volumes/kubernetes.io~projected/kube-api-access-vrk9g major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd1a2002-0045-4025-8c34-23ab182ca2a6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/bd1a2002-0045-4025-8c34-23ab182ca2a6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd1a2002-0045-4025-8c34-23ab182ca2a6/volumes/kubernetes.io~projected/kube-api-access-l9876:{mountpoint:/var/lib/kubelet/pods/bd1a2002-0045-4025-8c34-23ab182ca2a6/volumes/kubernetes.io~projected/kube-api-access-l9876 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd1a2002-0045-4025-8c34-23ab182ca2a6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/bd1a2002-0045-4025-8c34-23ab182ca2a6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d/volumes/kubernetes.io~projected/kube-api-access-44xcr:{mountpoint:/var/lib/kubelet/pods/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d/volumes/kubernetes.io~projected/kube-api-access-44xcr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2edb037-173f-4c27-987a-72d43df7fafb/volumes/kubernetes.io~projected/kube-api-access-6xcf5:{mountpoint:/var/lib/kubelet/pods/d2edb037-173f-4c27-987a-72d43df7fafb/volumes/kubernetes.io~projected/kube-api-access-6xcf5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6557aa0-6639-40cd-baed-ea6bf194348c/volumes/kubernetes.io~projected/kube-api-access-5k2qj:{mountpoint:/var/lib/kubelet/pods/d6557aa0-6639-40cd-baed-ea6bf194348c/volumes/kubernetes.io~projected/kube-api-access-5k2qj major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6557aa0-6639-40cd-baed-ea6bf194348c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d6557aa0-6639-40cd-baed-ea6bf194348c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dee312cb-ced0-47eb-8050-58bc39e462f6/volumes/kubernetes.io~projected/kube-api-access-nzsf2:{mountpoint:/var/lib/kubelet/pods/dee312cb-ced0-47eb-8050-58bc39e462f6/volumes/kubernetes.io~projected/kube-api-access-nzsf2 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2f5e019-7215-4c3d-8144-a32a9bba9965/volumes/kubernetes.io~projected/kube-api-access-xg582:{mountpoint:/var/lib/kubelet/pods/e2f5e019-7215-4c3d-8144-a32a9bba9965/volumes/kubernetes.io~projected/kube-api-access-xg582 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2f5e019-7215-4c3d-8144-a32a9bba9965/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/e2f5e019-7215-4c3d-8144-a32a9bba9965/volumes/kubernetes.io~secret/signing-key major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2c8f0b4-b6a7-4055-abca-b49923505435/volumes/kubernetes.io~projected/kube-api-access-vrlz4:{mountpoint:/var/lib/kubelet/pods/f2c8f0b4-b6a7-4055-abca-b49923505435/volumes/kubernetes.io~projected/kube-api-access-vrlz4 major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8b2aba4cc5fad6d2dca1cba0eaf8865f9be940c992deac1d726019b0b2445c22/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e071d72092d9e747fae7603d70e525cf3f61ab6c56efb8a40e6b3175fc0a403b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/23a0551fde4758eb9ce11bfeaa60f46dbdd1e92a57e49fab7d7df1fc0f48c94b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f601b520ec35ec0a5aca2d229d5dc6f9cda3026b710bf8f739ce3e50797cfc13/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dfdbba645e66c173321f02db2ebf778dbe3424218dbcfc5eb0b7bb87c019797b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d78483704831b7315888e77e13316f65614478fb32b127a136227f97ef75f70a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fc5ca4284fda711a9689c6123bf24ec882e24bfdfbe969fffc28725aa55a9836/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c741950f7c10e5421d6c5fb0ce558b7cad7fd14ac531f74e69261d8dd59c8825/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7c5d32667a3a22c5c1e1d276fd8beb89cd2e7c8adb0dcfc5756ec3ccc56f599b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/15bb6a7656f68f5a36920acb74d6146391242206ecb680cbb9dc8eb6b7f78cda/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/0ee033fd1ca344a1333a9ef970f2b5487e60a9e9bf95de7321705028f67ebfa1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/806e7de5d021c191ed59de8111f0c9a8a4e93ddf0f93be7048dc9ead4e4cc122/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/82fc339dc0b2ec392e17142c429e95e16304ab95c511e75b18ea6109aab7a236/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/105dd74739a404f995e99637778d841aa60b2e9826c43b4e832272a54a95fe46/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/27a2f236b99561bde13d73b5261b89104d9d0642fed642fc90edfa2cdd39a7cf/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/54693ae6acfa47aa4df43eda0f4492e1f22c004ea7d31e8591e070cccb795c19/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/dc1efdfefd61772f8c00b564212571762a7129d41e9faba157ffc3872071a766/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9ffc43cb0f564e0867c6fded860ef63c191a3dac27c853ea204c7cd98579abeb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d2a93a06676facdaf470d9a0e3a97f14a40d4a7912a89afb6ff5e0dc4ce3248f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/389dddab649d7eafedcbc19feaf5fef6ded489b43da2d6747a5a34678e98c5f6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/baa4375797510a86a35c83a7f041c289a9ea687e85ea0c12c0c14c98f82c7c2b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/b0f70550fef497d8cd0c9b2727fde42aff5e0177fe778145767ebeafee8c2732/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c7d1f4c2f2e0ca3f67610c74e58b91885cb8836dbc54b61b6d005ed55e897c58/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7764ee3f6016dcf576930d209d3a5d0dcae317bed7a463683ff43965d6709f6d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/4420721667eed66087336f2b18cfaec36414b01fdd6d9081c92eff61d19c7597/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/4ccd5cf0b4b5bdc26d9788f38ce72dc2649473f636d73a4cae714780d7675826/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/ce9622306dfbc9ff7e0c1fbf842e7fb617c025c51da94c58d5b26501028c1a98/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/8e2b32f0c290d27a73c13f322f431e5edc7e021a510948b486163a79f971814a/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/eb924235c50f63406b545ec16c7b404ae67640a73c33a44f71eff7f78113083a/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/a6cd2eb7f3b3b96a0397ac3d28e7e3480c46c31d6fee7ed1f80a693177ab89c1/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8eb6702a725a68d76e7a5b511985e7f8342a1bd22e25ceb0542d09533c7f2792/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/fb66cbc58fc4543b03e70c5975cac746d9b046b2222f9c189966aafa532e66ce/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a229433439bc766a4eb40338218f88cec27eed7ba6b988ca91e1ce489b850c19/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/dbbae769aadbf556a4da3147b8801d8530cdc46aa5f47db23445096388ff1ff5/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ddb9eb4237d689e7110e0d90c4d9edbb1bc4cff940d0983b1720b181e4915327/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ac709eb2696361837c8658a247888a4e5b90c3ab2784fffe5a6b56e84e953286/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/af6c520e5bc0a77e521251fc4142b2284a183ff300452a481b39594147c22a2c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/05bd4f8ba84e52be5eccacc1c491640bb5e0d299f80b3c350dc4a00afaa32e1e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/2e6d8c5e0dfe98d6679b7bd72ccaf888139bbbf5777317ca4501fd41ae9d5159/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.289580 4431 manager.go:217] Machine: {Timestamp:2026-10-30 05:53:11.289125394 +0000 UTC m=+0.095647132 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3124513 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20894e3385c55d85aec0c625373e30 SystemUUID:ec20894e-3385-c55d-85ae-c0c625373e30 BootID:f602cb3c-e710-4f4a-b171-f8fef0b62d50 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d68842197a388956e4be93267daa955a701e20d75801d34e8b4f24a7af19132/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d/volumes/kubernetes.io~projected/kube-api-access-44xcr DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d35b8efe205decd9dfa30e4ee3cf8d650ef51fedd0770cc1bc4801644abb193/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/210e30a3bb57bb25d3530b98b56362edec0d72cc3cada4f0104c292ca19aefb4/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a975ec4d-2d0d-4c74-ba0b-56a9dee81442/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1130b8d0ccbe242decea2dde9a37c1420b2a81304c99373cb746daf6d20774a4/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5397819fe4d9189ccd1df5648f17d45425d8136716feabf5b4186dc04b18842e/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/719219675a61628d482afec05c9300cc099f35136c0eb43dc3214eaa6a280f83/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/507dc60e78c61cf1f15dab1770255d1e2c60543b1cdaaa92847a762be1f49a34/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2f5e019-7215-4c3d-8144-a32a9bba9965/volumes/kubernetes.io~projected/kube-api-access-xg582 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9543ca89-5c01-4c07-9992-534b4955e331/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c/volumes/kubernetes.io~projected/kube-api-access-2fkcs DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1e19de3c90fb4f9fce255a3e1c8788b0cc04eeccbe08ea7d3b24057e22e2799/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd1a2002-0045-4025-8c34-23ab182ca2a6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f2c8f0b4-b6a7-4055-abca-b49923505435/volumes/kubernetes.io~projected/kube-api-access-vrlz4 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5705935-3763-404f-a84c-abc0133161bd/volumes/kubernetes.io~projected/kube-api-access-vrk9g DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0db0787edbd0cbce9e84487262579de70748640cfe4dc9bb968f8d23e0ce8807/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af2424601a36112818c4c1d61c7fa61a5cfb6671539dbca51cde3b0552cecf13/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd1a2002-0045-4025-8c34-23ab182ca2a6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f549965acef59f07eb50a042f648b8e4c483d466fdb8322854e3e42ac59abfb7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2edb037-173f-4c27-987a-72d43df7fafb/volumes/kubernetes.io~projected/kube-api-access-6xcf5 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6557aa0-6639-40cd-baed-ea6bf194348c/volumes/kubernetes.io~projected/kube-api-access-5k2qj DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/099ea3b3-43c1-4bbf-81aa-b3cbd803df66/volumes/kubernetes.io~projected/kube-api-access-dhsr4 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d5757a57b39ee79973ecca2f9b79bf2998c1ada9b75cf9d3b2770a059a8fb5c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6557aa0-6639-40cd-baed-ea6bf194348c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9645f0dc5f838c70bdac34995724a404c3095df1207e41fe6b92ce7980fd36f/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/981bde37bab66f1ab27f4830eb9b63bde7078e6ff1d2cc10f06ff03c02380c8f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15d1e924-1f5b-4867-aa83-cd43814eebbb/volumes/kubernetes.io~projected/kube-api-access-gswk6 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e2f5e019-7215-4c3d-8144-a32a9bba9965/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/a98d05d0-c37f-4a97-b943-fd4b728e29d0/volumes/kubernetes.io~projected/kube-api-access-bcsk8 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6/volumes/kubernetes.io~projected/kube-api-access-zzqlk DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd1a2002-0045-4025-8c34-23ab182ca2a6/volumes/kubernetes.io~projected/kube-api-access-l9876 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8/volumes/kubernetes.io~projected/kube-api-access-7q88h DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dee312cb-ced0-47eb-8050-58bc39e462f6/volumes/kubernetes.io~projected/kube-api-access-nzsf2 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:5397819fe4d9189 MacAddress:96:79:d0:8a:b5:85 Speed:10000 Mtu:8901} {Name:719219675a61628 MacAddress:6e:e4:93:11:d1:c3 Speed:10000 Mtu:8901} {Name:a9645f0dc5f838c MacAddress:9a:ad:c5:f9:ab:ca Speed:10000 Mtu:8901} {Name:af2424601a36112 MacAddress:8a:29:44:42:16:d9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d7:f6:49:81 Speed:0 Mtu:9001} {Name:br-int MacAddress:c6:de:ba:44:b6:1f Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d7:f6:49:81 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:5d:d6:3b:60:b6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d2:9b:67:bf:c3:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.289833 4431 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.289873 4431 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290162 4431 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290187 4431 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290324 4431 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290333 4431 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290347 4431 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290359 4431 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290423 4431 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290577 4431 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290682 4431 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290717 4431 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290726 4431 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290736 4431 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290744 4431 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.290752 4431 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.291353 4431 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.291364 4431 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:53:11.295254 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.292011 4431 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295359 4431 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295374 4431 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295486 4431 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295733 4431 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295774 4431 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295885 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295897 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295902 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295908 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:53:11.295912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295914 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:53:11.296166 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295923 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:53:11.296166 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295933 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:53:11.296166 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295940 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:53:11.296166 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295947 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:53:11.296166 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295954 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:53:11.296166 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295962 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:53:11.296166 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.295971 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:53:11.296166 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.296001 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:53:11.296166 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.296009 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:53:11.299284 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.299271 4431 server.go:1263] "Started kubelet" Oct 30 05:53:11.299396 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.299368 4431 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:53:11.299450 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.299380 4431 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:53:11.299783 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.299762 4431 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:53:11.299823 ip-10-0-136-123 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:53:11.300084 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.300047 4431 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:53:11.302741 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.301637 4431 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:53:11.302741 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.301758 4431 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:53:11.302741 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.301767 4431 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:53:11.302741 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.301842 4431 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:53:11.302741 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.301974 4431 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:53:11.302741 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.302209 4431 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:53:11.302741 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.302260 4431 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:47:33 +0000 UTC" deadline="2028-07-05 11:20:01.381548453 +0000 UTC" Oct 30 05:53:11.302741 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.302291 4431 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14741h26m50.079260253s" Oct 30 05:53:11.302741 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:11.301781 4431 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-123.ec2.internal\" not found" Oct 30 05:53:11.303328 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303289 4431 factory.go:55] Registering systemd factory Oct 30 05:53:11.303409 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303332 4431 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:53:11.303468 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303297 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d26977a-058d-4e3b-9b5b-f43a0e66fe3c" volumeName="kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-tuned" seLinuxMountContext="" Oct 30 05:53:11.303468 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303438 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a975ec4d-2d0d-4c74-ba0b-56a9dee81442" volumeName="kubernetes.io/configmap/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-konnectivity-ca" seLinuxMountContext="" Oct 30 05:53:11.303468 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303453 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a98d05d0-c37f-4a97-b943-fd4b728e29d0" volumeName="kubernetes.io/projected/a98d05d0-c37f-4a97-b943-fd4b728e29d0-kube-api-access-bcsk8" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303468 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2f5e019-7215-4c3d-8144-a32a9bba9965" volumeName="kubernetes.io/projected/e2f5e019-7215-4c3d-8144-a32a9bba9965-kube-api-access-xg582" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303480 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9543ca89-5c01-4c07-9992-534b4955e331" volumeName="kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303491 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="099ea3b3-43c1-4bbf-81aa-b3cbd803df66" volumeName="kubernetes.io/projected/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-kube-api-access-dhsr4" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303501 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a975ec4d-2d0d-4c74-ba0b-56a9dee81442" volumeName="kubernetes.io/secret/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-agent-certs" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303513 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1a2002-0045-4025-8c34-23ab182ca2a6" volumeName="kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-script-lib" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303530 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6557aa0-6639-40cd-baed-ea6bf194348c" volumeName="kubernetes.io/secret/d6557aa0-6639-40cd-baed-ea6bf194348c-cert" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303543 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" volumeName="kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-binary-copy" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303564 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1a2002-0045-4025-8c34-23ab182ca2a6" volumeName="kubernetes.io/secret/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303580 4431 factory.go:153] Registering CRI-O factory Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303587 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87c9e48-d271-4c0c-92d3-38fbdac9ab2d" volumeName="kubernetes.io/projected/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-kube-api-access-44xcr" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303601 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c8f0b4-b6a7-4055-abca-b49923505435" volumeName="kubernetes.io/configmap/f2c8f0b4-b6a7-4055-abca-b49923505435-config-volume" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303613 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" volumeName="kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 05:53:11.303630 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303627 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" volumeName="kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303639 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d1e924-1f5b-4867-aa83-cd43814eebbb" volumeName="kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303651 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d1e924-1f5b-4867-aa83-cd43814eebbb" volumeName="kubernetes.io/projected/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-api-access-gswk6" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303662 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5705935-3763-404f-a84c-abc0133161bd" volumeName="kubernetes.io/configmap/b5705935-3763-404f-a84c-abc0133161bd-iptables-alerter-script" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303673 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6557aa0-6639-40cd-baed-ea6bf194348c" volumeName="kubernetes.io/projected/d6557aa0-6639-40cd-baed-ea6bf194348c-kube-api-access-5k2qj" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303683 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2f5e019-7215-4c3d-8144-a32a9bba9965" volumeName="kubernetes.io/secret/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-key" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303694 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c8f0b4-b6a7-4055-abca-b49923505435" volumeName="kubernetes.io/projected/f2c8f0b4-b6a7-4055-abca-b49923505435-kube-api-access-vrlz4" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303710 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" volumeName="kubernetes.io/projected/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-kube-api-access-7q88h" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303721 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d26977a-058d-4e3b-9b5b-f43a0e66fe3c" volumeName="kubernetes.io/projected/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-kube-api-access-2fkcs" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303594 4431 factory.go:223] Registration of the crio container factory successfully Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303732 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a98d05d0-c37f-4a97-b943-fd4b728e29d0" volumeName="kubernetes.io/configmap/a98d05d0-c37f-4a97-b943-fd4b728e29d0-serviceca" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303745 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5705935-3763-404f-a84c-abc0133161bd" volumeName="kubernetes.io/projected/b5705935-3763-404f-a84c-abc0133161bd-kube-api-access-vrk9g" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303757 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87c9e48-d271-4c0c-92d3-38fbdac9ab2d" volumeName="kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cni-binary-copy" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303768 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2f5e019-7215-4c3d-8144-a32a9bba9965" volumeName="kubernetes.io/configmap/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-cabundle" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303800 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="099ea3b3-43c1-4bbf-81aa-b3cbd803df66" volumeName="kubernetes.io/empty-dir/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-tmp-dir" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303815 4431 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303851 4431 factory.go:103] Registering Raw factory Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303816 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d1e924-1f5b-4867-aa83-cd43814eebbb" volumeName="kubernetes.io/empty-dir/15d1e924-1f5b-4867-aa83-cd43814eebbb-data-volume" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303901 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a4476b6-e86f-49cd-a813-8cffdfcc3ca6" volumeName="kubernetes.io/projected/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-kube-api-access-zzqlk" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303889 4431 manager.go:1199] Started watching for new ooms in manager Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303918 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87c9e48-d271-4c0c-92d3-38fbdac9ab2d" volumeName="kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-daemon-config" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303934 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d26977a-058d-4e3b-9b5b-f43a0e66fe3c" volumeName="kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-tmp" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303947 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1a2002-0045-4025-8c34-23ab182ca2a6" volumeName="kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-env-overrides" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303960 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1a2002-0045-4025-8c34-23ab182ca2a6" volumeName="kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-config" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303971 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd1a2002-0045-4025-8c34-23ab182ca2a6" volumeName="kubernetes.io/projected/bd1a2002-0045-4025-8c34-23ab182ca2a6-kube-api-access-l9876" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303984 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2edb037-173f-4c27-987a-72d43df7fafb" volumeName="kubernetes.io/projected/d2edb037-173f-4c27-987a-72d43df7fafb-kube-api-access-6xcf5" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.303995 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dee312cb-ced0-47eb-8050-58bc39e462f6" volumeName="kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.304008 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c8f0b4-b6a7-4055-abca-b49923505435" volumeName="kubernetes.io/empty-dir/f2c8f0b4-b6a7-4055-abca-b49923505435-tmp-dir" seLinuxMountContext="" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.304021 4431 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:53:11.304259 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.304029 4431 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:53:11.305602 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.304390 4431 manager.go:319] Starting recovery of all containers Oct 30 05:53:11.328064 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.327940 4431 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:53:11.329692 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.329664 4431 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:53:11.329692 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.329692 4431 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:53:11.329855 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.329715 4431 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:53:11.329855 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:11.329761 4431 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 05:53:11.344477 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.344452 4431 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:11.354161 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.354142 4431 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 05:53:11.354575 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.354547 4431 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="bb7c91c48170e1dded3d7980f47e922e9a0f4b41d3c28e7d13b901ab09ed20e6" exitCode=0 Oct 30 05:53:11.354575 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.354572 4431 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="0dc0854da946bedf9fcf0a77144079108d707575b8e8e8ad9abfa8ef51e1f951" exitCode=0 Oct 30 05:53:11.354714 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.354581 4431 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="d7f19f90653557b5f7d19b1a9e493b64b9ffc148e788faf836889eda70689a71" exitCode=0 Oct 30 05:53:11.354714 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.354590 4431 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="6738469b21b01c1799facc65579a403eda34caf530f6832c0923c9e69ba931b0" exitCode=0 Oct 30 05:53:11.354714 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.354597 4431 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="b796259a12d9c90b68d6e9c74587fa68664f351ca6dc0e182093c2b269604c1d" exitCode=0 Oct 30 05:53:11.354714 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.354606 4431 generic.go:356] "Generic (PLEG): container finished" podID="10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8" containerID="a1133cb791e075085b2bedbe291447a6fff7cf54f6a201a34d7d40cbcd0e5281" exitCode=0 Oct 30 05:53:11.357040 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.357018 4431 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:11.364368 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.364346 4431 generic.go:356] "Generic (PLEG): container finished" podID="7c8a05fdf5abb06be0c4a4fee3879794" containerID="e53d6128d59f87946bce79f37a156c87b364da02642e5c51dad0f5e3b86c67f0" exitCode=0 Oct 30 05:53:11.369373 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.369353 4431 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:11.369467 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.369379 4431 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:11.373618 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.373602 4431 manager.go:324] Recovery completed Oct 30 05:53:11.386477 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386423 4431 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:53:11.386477 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386437 4431 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:53:11.386477 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386455 4431 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:53:11.386663 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386601 4431 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 05:53:11.386663 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386617 4431 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 05:53:11.386663 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386640 4431 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 05:53:11.386663 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386648 4431 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 05:53:11.386663 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386656 4431 policy_none.go:50] "Start" Oct 30 05:53:11.386897 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386667 4431 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:53:11.386897 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386680 4431 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:53:11.386897 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386847 4431 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 05:53:11.386897 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386858 4431 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 05:53:11.386897 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.386866 4431 policy_none.go:44] "Start" Oct 30 05:53:11.390802 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.390776 4431 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:53:11.393310 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.393290 4431 manager.go:341] "Starting Device Plugin manager" Oct 30 05:53:11.393960 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:11.393333 4431 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:53:11.393960 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.393961 4431 server.go:87] "Starting device plugin registration server" Oct 30 05:53:11.394226 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.394215 4431 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:53:11.394267 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.394227 4431 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:53:11.394385 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.394336 4431 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:53:11.394497 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.394443 4431 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:53:11.394551 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.394512 4431 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:53:11.394551 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.394522 4431 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:53:11.394645 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.394581 4431 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:53:11.394496279Z","UUID":"e31571a5-e4ef-4158-aa1e-aa5349dfee98","Handler":null,"Name":"","Endpoint":""} Oct 30 05:53:11.395933 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:11.395917 4431 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:53:11.396036 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.395984 4431 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:53:11.396036 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.396011 4431 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:53:11.430212 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.430177 4431 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"] Oct 30 05:53:11.430689 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.430652 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" event={"ID":"cc0cc40e9c1a761dcecd1aa2a7591d53","Type":"ContainerStarted","Data":"2ea61acd64e79a29e6c7c8bf2cd01054969e38199cbe4448480a4099c0906d71"} Oct 30 05:53:11.430783 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.430693 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" event={"ID":"cc0cc40e9c1a761dcecd1aa2a7591d53","Type":"ContainerStarted","Data":"f549965acef59f07eb50a042f648b8e4c483d466fdb8322854e3e42ac59abfb7"} Oct 30 05:53:11.430783 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.430722 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" event={"ID":"7c8a05fdf5abb06be0c4a4fee3879794","Type":"ContainerStarted","Data":"8ae5c9a6f8ae4b1ffc0a7421c488a7f4a4e0887e441306809850fdc0c0851276"} Oct 30 05:53:11.430783 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.430731 4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.430895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.430735 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" event={"ID":"7c8a05fdf5abb06be0c4a4fee3879794","Type":"ContainerDied","Data":"e53d6128d59f87946bce79f37a156c87b364da02642e5c51dad0f5e3b86c67f0"} Oct 30 05:53:11.430895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.430825 4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.430895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.430831 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" event={"ID":"7c8a05fdf5abb06be0c4a4fee3879794","Type":"ContainerStarted","Data":"6d68842197a388956e4be93267daa955a701e20d75801d34e8b4f24a7af19132"} Oct 30 05:53:11.453928 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.453909 4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:11.453971 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:11.453954 4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.454604 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.454589 4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:11.454661 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:11.454624 4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.494738 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.494719 4431 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:53:11.495816 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.495802 4431 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:53:11.495876 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.495828 4431 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:53:11.495876 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.495837 4431 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-123.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:53:11.495876 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.495862 4431 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.532892 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.532877 4431 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.532962 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.532921 4431 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.554451 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.554434 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.554528 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.554456 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.554528 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.554474 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc0cc40e9c1a761dcecd1aa2a7591d53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" (UID: \"cc0cc40e9c1a761dcecd1aa2a7591d53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.655645 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.655585 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.655645 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.655537 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.655645 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.655639 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc0cc40e9c1a761dcecd1aa2a7591d53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" (UID: \"cc0cc40e9c1a761dcecd1aa2a7591d53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.655798 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.655684 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc0cc40e9c1a761dcecd1aa2a7591d53-config\") pod \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" (UID: \"cc0cc40e9c1a761dcecd1aa2a7591d53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.655798 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.655692 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:11.655798 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:11.655717 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c8a05fdf5abb06be0c4a4fee3879794-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" (UID: \"7c8a05fdf5abb06be0c4a4fee3879794\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:12.291519 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.291486 4431 apiserver.go:52] "Watching apiserver" Oct 30 05:53:12.332076 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.332050 4431 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:53:12.332576 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.332558 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-b4v22","openshift-insights/insights-runtime-extractor-rmk2m","kube-system/konnectivity-agent-kf896","openshift-dns/dns-default-7b2h7","openshift-multus/multus-2fn7x","openshift-ovn-kubernetes/ovnkube-node-8zb4l","openshift-cluster-node-tuning-operator/tuned-9qrxj","openshift-ingress-canary/ingress-canary-jczv9","openshift-multus/multus-additional-cni-plugins-82lcq","openshift-network-diagnostics/network-check-target-2jjjp","kube-system/global-pull-secret-syncer-lbqgc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj","openshift-multus/network-metrics-daemon-8ntvq","openshift-image-registry/node-ca-v2swz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal","openshift-service-ca/service-ca-5fffc97bdf-26ms6","openshift-network-operator/iptables-alerter-cnmml","kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal"] Oct 30 05:53:12.338569 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.338546 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:12.339757 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.339742 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:12.340136 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.340122 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.348469 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.348444 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.348469 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.348456 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-27cwd\"" Oct 30 05:53:12.348603 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.348448 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.348603 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.348485 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.348603 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.348499 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.352538 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.352512 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:53:12.352609 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.352594 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mpzcb\"" Oct 30 05:53:12.358687 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.358673 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.358739 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.358691 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.358739 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.358725 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.358812 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.358804 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:53:12.358858 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.358829 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:53:12.358858 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.358831 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:53:12.358858 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.358848 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.358949 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.358854 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.359962 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.359946 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:53:12.360003 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.359967 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.360047 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.360039 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:53:12.362750 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.362734 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mkzdc\"" Oct 30 05:53:12.362818 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.362804 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q46lp\"" Oct 30 05:53:12.362872 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.362859 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d9dv5\"" Oct 30 05:53:12.362917 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.362904 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l4lh5\"" Oct 30 05:53:12.362955 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.362934 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:53:12.362955 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.362939 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:53:12.363020 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.362958 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5hkvn\"" Oct 30 05:53:12.363020 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.362968 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q2ncw\"" Oct 30 05:53:12.363020 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.363009 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vzk7w\"" Oct 30 05:53:12.363330 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.363318 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-99gsm\"" Oct 30 05:53:12.366128 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.366106 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:53:12.366218 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.366167 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:53:12.366373 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.366356 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:53:12.366440 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.366381 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 05:53:12.366489 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.366468 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:53:12.366732 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.366718 4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:53:12.366884 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.366871 4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:12.370886 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.370870 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:53:12.370946 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.370926 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6cnz\"" Oct 30 05:53:12.370998 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.370954 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x8h66\"" Oct 30 05:53:12.370998 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.370970 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:53:12.370998 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.370978 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nrzkq\"" Oct 30 05:53:12.371243 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.371226 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.371327 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.371237 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-plsgv\"" Oct 30 05:53:12.371327 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.371285 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.371408 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.371347 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:53:12.371408 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.371361 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntdkp\"" Oct 30 05:53:12.371408 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.371369 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 05:53:12.371408 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.371348 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.371408 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.371402 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.372252 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372211 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:53:12.372341 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372258 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:53:12.372341 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372257 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:53:12.372341 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372287 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.372341 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372296 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.372341 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372338 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:53:12.372510 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372347 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.372510 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372363 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xln8v\"" Oct 30 05:53:12.372510 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372404 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.372510 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372448 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.372510 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372496 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:53:12.372750 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372511 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.372750 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372530 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:53:12.372750 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.372544 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:12.390677 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.390660 4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:12.390743 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:12.390694 4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-123.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-123.ec2.internal" Oct 30 05:53:12.391317 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.391289 4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:12.391365 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:12.391342 4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-123.ec2.internal" Oct 30 05:53:12.403369 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.403350 4431 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:53:12.458680 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458655 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:12.458766 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458683 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.458766 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458703 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-kubelet\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.458766 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458719 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.458766 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458734 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-bin\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.458766 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458750 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovn-node-metrics-cert\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.458981 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458825 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.458981 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458874 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.458981 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458900 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-multus\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.459117 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.458981 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-k8s-cni-cncf-io\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.459117 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459026 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.459117 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459058 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-tuned\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.459117 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459088 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-registration-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.459117 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459111 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459132 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459136 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459159 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-tuned\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459168 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c8f0b4-b6a7-4055-abca-b49923505435-tmp-dir\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459198 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-hosts-file\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459217 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-hostroot\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459235 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2c8f0b4-b6a7-4055-abca-b49923505435-tmp-dir\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459244 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-sys\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459263 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-env-overrides\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459291 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-bin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.459347 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459327 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovn-node-metrics-cert\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.459642 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459418 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6557aa0-6639-40cd-baed-ea6bf194348c-cert\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:12.459642 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459430 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-env-overrides\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.459642 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459450 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsk8\" (UniqueName: \"kubernetes.io/projected/a98d05d0-c37f-4a97-b943-fd4b728e29d0-kube-api-access-bcsk8\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:53:12.459642 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459562 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:12.459642 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459579 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6557aa0-6639-40cd-baed-ea6bf194348c-cert\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:12.459808 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459651 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.459808 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459734 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-etc-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.459808 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459769 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-netd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.459808 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459799 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-script-lib\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.459920 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459833 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.459920 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459898 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9543ca89-5c01-4c07-9992-534b4955e331-original-pull-secret\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:12.459973 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459929 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-netns\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.460006 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.459990 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-script-lib\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.460035 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460011 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xcr\" (UniqueName: \"kubernetes.io/projected/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-kube-api-access-44xcr\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.460066 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460042 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.460066 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460050 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-lib-modules\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.460156 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460141 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkcs\" (UniqueName: \"kubernetes.io/projected/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-kube-api-access-2fkcs\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.460191 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460173 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a98d05d0-c37f-4a97-b943-fd4b728e29d0-host\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:53:12.460228 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460213 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-sys-fs\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.460274 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460261 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xcf5\" (UniqueName: \"kubernetes.io/projected/d2edb037-173f-4c27-987a-72d43df7fafb-kube-api-access-6xcf5\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.460321 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460291 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-system-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.460358 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460328 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-modprobe-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.460358 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460347 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.460451 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460435 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9876\" (UniqueName: \"kubernetes.io/projected/bd1a2002-0045-4025-8c34-23ab182ca2a6-kube-api-access-l9876\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.460488 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460477 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-kubernetes\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.460517 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460504 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-var-lib-kubelet\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.460615 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460604 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-tmp-dir\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:53:12.460683 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460668 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-tmp-dir\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:53:12.460715 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460701 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-system-cni-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.460744 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460726 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cnibin\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.460778 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460745 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrk9g\" (UniqueName: \"kubernetes.io/projected/b5705935-3763-404f-a84c-abc0133161bd-kube-api-access-vrk9g\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:53:12.460778 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460771 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-dbus\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:12.460834 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460785 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-etc-kubernetes\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.460895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460882 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsr4\" (UniqueName: \"kubernetes.io/projected/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-kube-api-access-dhsr4\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:53:12.460941 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460926 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-socket-dir-parent\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.460979 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.460964 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5705935-3763-404f-a84c-abc0133161bd-host-slash\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:53:12.461013 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461001 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2qj\" (UniqueName: \"kubernetes.io/projected/d6557aa0-6639-40cd-baed-ea6bf194348c-kube-api-access-5k2qj\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:12.461042 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461030 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-device-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.461072 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461064 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15d1e924-1f5b-4867-aa83-cd43814eebbb-crio-socket\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.461150 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461132 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15d1e924-1f5b-4867-aa83-cd43814eebbb-data-volume\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.461178 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461162 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5705935-3763-404f-a84c-abc0133161bd-iptables-alerter-script\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:53:12.461241 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461180 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-kubelet-config\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:12.461241 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461197 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-slash\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.461294 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461241 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.461294 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461278 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c8f0b4-b6a7-4055-abca-b49923505435-config-volume\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:12.461368 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461316 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-conf-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.461429 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461409 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-daemon-config\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.461464 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461440 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15d1e924-1f5b-4867-aa83-cd43814eebbb-data-volume\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.461464 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461451 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5705935-3763-404f-a84c-abc0133161bd-iptables-alerter-script\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:53:12.461558 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461476 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-socket-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.461558 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461491 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c8f0b4-b6a7-4055-abca-b49923505435-config-volume\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:12.461640 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461554 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.461640 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461582 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gswk6\" (UniqueName: \"kubernetes.io/projected/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-api-access-gswk6\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.461640 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461618 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-kubelet\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.461780 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461701 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-systemd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.461780 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461734 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-node-log\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.461780 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461738 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-daemon-config\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.461780 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461763 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-cabundle\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:12.461936 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461803 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg582\" (UniqueName: \"kubernetes.io/projected/e2f5e019-7215-4c3d-8144-a32a9bba9965-kube-api-access-xg582\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:12.461936 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461822 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-konnectivity-ca\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:53:12.461936 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461856 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:12.461936 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461935 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrlz4\" (UniqueName: \"kubernetes.io/projected/f2c8f0b4-b6a7-4055-abca-b49923505435-kube-api-access-vrlz4\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:12.462129 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.461984 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.462129 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462014 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-systemd-units\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.462129 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462015 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-cabundle\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:12.462129 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462033 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-konnectivity-ca\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:53:12.462129 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462061 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-ovn\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.462129 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462082 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-agent-certs\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:53:12.462129 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462112 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqlk\" (UniqueName: \"kubernetes.io/projected/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-kube-api-access-zzqlk\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462149 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-key\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462172 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-conf\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462235 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-run\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462281 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-host\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462326 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-config\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462334 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a975ec4d-2d0d-4c74-ba0b-56a9dee81442-agent-certs\") pod \"konnectivity-agent-kf896\" (UID: \"a975ec4d-2d0d-4c74-ba0b-56a9dee81442\") " pod="kube-system/konnectivity-agent-kf896" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462351 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462380 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysconfig\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462412 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2f5e019-7215-4c3d-8144-a32a9bba9965-signing-key\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:12.462474 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462476 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-netns\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462500 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a98d05d0-c37f-4a97-b943-fd4b728e29d0-serviceca\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462511 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd1a2002-0045-4025-8c34-23ab182ca2a6-ovnkube-config\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462517 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q88h\" (UniqueName: \"kubernetes.io/projected/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-kube-api-access-7q88h\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462578 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cnibin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462611 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cni-binary-copy\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462650 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-multus-certs\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462679 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462707 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-var-lib-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462707 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a98d05d0-c37f-4a97-b943-fd4b728e29d0-serviceca\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462759 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cni-binary-copy\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462800 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-log-socket\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462831 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-systemd\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.462848 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462847 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-tmp\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.463193 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462851 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.463193 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.462927 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-tmp\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.480894 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.480864 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsk8\" (UniqueName: \"kubernetes.io/projected/a98d05d0-c37f-4a97-b943-fd4b728e29d0-kube-api-access-bcsk8\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:53:12.481345 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.481331 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xcf5\" (UniqueName: \"kubernetes.io/projected/d2edb037-173f-4c27-987a-72d43df7fafb-kube-api-access-6xcf5\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.481420 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.481404 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9876\" (UniqueName: \"kubernetes.io/projected/bd1a2002-0045-4025-8c34-23ab182ca2a6-kube-api-access-l9876\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.481555 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.481503 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xcr\" (UniqueName: \"kubernetes.io/projected/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-kube-api-access-44xcr\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.481555 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.481504 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkcs\" (UniqueName: \"kubernetes.io/projected/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-kube-api-access-2fkcs\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.486776 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.486759 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzsf2\" (UniqueName: \"kubernetes.io/projected/dee312cb-ced0-47eb-8050-58bc39e462f6-kube-api-access-nzsf2\") pod \"network-check-target-2jjjp\" (UID: \"dee312cb-ced0-47eb-8050-58bc39e462f6\") " pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:12.488729 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.488710 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrk9g\" (UniqueName: \"kubernetes.io/projected/b5705935-3763-404f-a84c-abc0133161bd-kube-api-access-vrk9g\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:53:12.488797 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.488732 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsr4\" (UniqueName: \"kubernetes.io/projected/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-kube-api-access-dhsr4\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:53:12.488912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.488897 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q88h\" (UniqueName: \"kubernetes.io/projected/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-kube-api-access-7q88h\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.489237 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.489218 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg582\" (UniqueName: \"kubernetes.io/projected/e2f5e019-7215-4c3d-8144-a32a9bba9965-kube-api-access-xg582\") pod \"service-ca-5fffc97bdf-26ms6\" (UID: \"e2f5e019-7215-4c3d-8144-a32a9bba9965\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26ms6" Oct 30 05:53:12.489278 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.489232 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2qj\" (UniqueName: \"kubernetes.io/projected/d6557aa0-6639-40cd-baed-ea6bf194348c-kube-api-access-5k2qj\") pod \"ingress-canary-jczv9\" (UID: \"d6557aa0-6639-40cd-baed-ea6bf194348c\") " pod="openshift-ingress-canary/ingress-canary-jczv9" Oct 30 05:53:12.491776 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.491758 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqlk\" (UniqueName: \"kubernetes.io/projected/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-kube-api-access-zzqlk\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:12.493495 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.493476 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gswk6\" (UniqueName: \"kubernetes.io/projected/15d1e924-1f5b-4867-aa83-cd43814eebbb-kube-api-access-gswk6\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.493555 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.493535 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrlz4\" (UniqueName: \"kubernetes.io/projected/f2c8f0b4-b6a7-4055-abca-b49923505435-kube-api-access-vrlz4\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:12.563268 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563215 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-socket-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.563268 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563242 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.563268 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563259 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-kubelet\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563273 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-systemd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563288 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-node-log\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563350 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563364 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-socket-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563374 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563366 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-kubelet\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563366 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-systemd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563380 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-node-log\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:12.563379 4431 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:12.563430 4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563436 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563460 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-systemd-units\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563478 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563407 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-systemd-units\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:12.563497 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:13.063480828 +0000 UTC m=+1.870002577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563513 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-ovn\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:12.563537 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls podName:f2c8f0b4-b6a7-4055-abca-b49923505435 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:13.063513032 +0000 UTC m=+1.870034761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls") pod "dns-default-7b2h7" (UID: "f2c8f0b4-b6a7-4055-abca-b49923505435") : secret "dns-default-metrics-tls" not found Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563554 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-ovn\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563588 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-conf\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563604 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-run\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563619 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-host\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563636 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563674 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-host\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563688 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-run\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:12.563711 4431 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563720 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-conf\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563719 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysconfig\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563749 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-netns\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:12.563759 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:13.063747328 +0000 UTC m=+1.870269064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : secret "metrics-daemon-secret" not found Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563761 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysconfig\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563770 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-netns\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563786 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cnibin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563807 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-multus-certs\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563823 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-var-lib-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563839 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-cnibin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563839 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-log-socket\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563864 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-systemd\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563866 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-multus-certs\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563886 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-var-lib-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563890 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-log-socket\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.563895 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563890 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-kubelet\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563923 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563929 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-kubelet\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563939 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-bin\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563957 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563923 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-systemd\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.563981 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-multus\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564008 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-run-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564027 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-bin\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564053 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-multus\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564069 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-k8s-cni-cncf-io\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564092 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564101 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-k8s-cni-cncf-io\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564109 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-registration-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564111 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564125 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564145 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-hosts-file\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564157 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-registration-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564168 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-hostroot\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564177 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564192 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-sys\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564217 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-sysctl-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564220 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-bin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564195 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/099ea3b3-43c1-4bbf-81aa-b3cbd803df66-hosts-file\") pod \"node-resolver-b4v22\" (UID: \"099ea3b3-43c1-4bbf-81aa-b3cbd803df66\") " pod="openshift-dns/node-resolver-b4v22" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564230 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-sys\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564244 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564253 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-var-lib-cni-bin\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564261 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-etc-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564277 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-netd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564221 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-hostroot\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564331 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-cni-netd\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564335 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564334 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-etc-openvswitch\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564356 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-netns\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564373 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-lib-modules\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564395 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a98d05d0-c37f-4a97-b943-fd4b728e29d0-host\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564412 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-host-run-netns\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564417 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-sys-fs\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564441 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-system-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564450 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a98d05d0-c37f-4a97-b943-fd4b728e29d0-host\") pod \"node-ca-v2swz\" (UID: \"a98d05d0-c37f-4a97-b943-fd4b728e29d0\") " pod="openshift-image-registry/node-ca-v2swz" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564465 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-modprobe-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564480 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-sys-fs\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564479 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-lib-modules\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564483 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564498 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-system-cni-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564518 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-etc-selinux\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564520 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-kubernetes\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564538 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-var-lib-kubelet\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564553 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-system-cni-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564570 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cnibin\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564573 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-kubernetes\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564582 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-var-lib-kubelet\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564586 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d26977a-058d-4e3b-9b5b-f43a0e66fe3c-etc-modprobe-d\") pod \"tuned-9qrxj\" (UID: \"9d26977a-058d-4e3b-9b5b-f43a0e66fe3c\") " pod="openshift-cluster-node-tuning-operator/tuned-9qrxj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564585 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-dbus\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564617 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-etc-kubernetes\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564620 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-system-cni-dir\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564628 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8-cnibin\") pod \"multus-additional-cni-plugins-82lcq\" (UID: \"10bfdc3b-a4a7-4fcc-aa43-90ade5eda8d8\") " pod="openshift-multus/multus-additional-cni-plugins-82lcq" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564633 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-socket-dir-parent\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564624 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-dbus\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564648 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5705935-3763-404f-a84c-abc0133161bd-host-slash\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564659 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-etc-kubernetes\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564693 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-socket-dir-parent\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564705 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-device-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564750 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5705935-3763-404f-a84c-abc0133161bd-host-slash\") pod \"iptables-alerter-cnmml\" (UID: \"b5705935-3763-404f-a84c-abc0133161bd\") " pod="openshift-network-operator/iptables-alerter-cnmml" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564754 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15d1e924-1f5b-4867-aa83-cd43814eebbb-crio-socket\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564765 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2edb037-173f-4c27-987a-72d43df7fafb-device-dir\") pod \"aws-ebs-csi-driver-node-jh4pj\" (UID: \"d2edb037-173f-4c27-987a-72d43df7fafb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jh4pj" Oct 30 05:53:12.564770 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564805 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15d1e924-1f5b-4867-aa83-cd43814eebbb-crio-socket\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:12.566565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564825 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-kubelet-config\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:12.566565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564841 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-slash\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.566565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564856 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.566565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564860 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9543ca89-5c01-4c07-9992-534b4955e331-kubelet-config\") pod \"global-pull-secret-syncer-lbqgc\" (UID: \"9543ca89-5c01-4c07-9992-534b4955e331\") " pod="kube-system/global-pull-secret-syncer-lbqgc" Oct 30 05:53:12.566565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564870 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-conf-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:12.566565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564885 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-slash\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.566565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564893 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd1a2002-0045-4025-8c34-23ab182ca2a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8zb4l\" (UID: \"bd1a2002-0045-4025-8c34-23ab182ca2a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:12.566565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:12.564906 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c87c9e48-d271-4c0c-92d3-38fbdac9ab2d-multus-conf-dir\") pod \"multus-2fn7x\" (UID: \"c87c9e48-d271-4c0c-92d3-38fbdac9ab2d\") " pod="openshift-multus/multus-2fn7x" Oct 30 05:53:13.068679 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:13.068656 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:13.068762 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:13.068694 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:13.068762 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:13.068718 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:13.068828 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:13.068799 4431 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:13.068828 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:13.068802 4431 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:13.068881 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:13.068823 4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:13.068881 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:13.068855 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:14.068840017 +0000 UTC m=+2.875361743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : secret "metrics-daemon-secret" not found Oct 30 05:53:13.068881 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:13.068870 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:14.068864479 +0000 UTC m=+2.875386205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:13.068972 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:13.068883 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls podName:f2c8f0b4-b6a7-4055-abca-b49923505435 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:14.068876641 +0000 UTC m=+2.875398367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls") pod "dns-default-7b2h7" (UID: "f2c8f0b4-b6a7-4055-abca-b49923505435") : secret "dns-default-metrics-tls" not found Oct 30 05:53:13.759844 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:13.759816 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:14.076529 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:14.076453 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:14.076529 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:14.076496 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:14.076736 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:14.076561 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:14.076736 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:14.076577 4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:53:14.076736 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:14.076635 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls podName:f2c8f0b4-b6a7-4055-abca-b49923505435 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:16.076622201 +0000 UTC m=+4.883143931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls") pod "dns-default-7b2h7" (UID: "f2c8f0b4-b6a7-4055-abca-b49923505435") : secret "dns-default-metrics-tls" not found Oct 30 05:53:14.076736 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:14.076644 4431 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:14.076736 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:14.076701 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:16.076688377 +0000 UTC m=+4.883210120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : secret "metrics-daemon-secret" not found Oct 30 05:53:14.076736 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:14.076648 4431 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:53:14.076736 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:14.076732 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls podName:15d1e924-1f5b-4867-aa83-cd43814eebbb nodeName:}" failed. No retries permitted until 2026-10-30 05:53:16.076726348 +0000 UTC m=+4.883248075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rmk2m" (UID: "15d1e924-1f5b-4867-aa83-cd43814eebbb") : secret "insights-runtime-extractor-tls" not found Oct 30 05:53:14.862509 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:14.862480 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:14.862945 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:14.862677 4431 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:53:15.028774 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:15.028747 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8zb4l" Oct 30 05:53:16.089843 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.089811 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:16.089843 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.089851 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:16.090391 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.089870 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:16.090391 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:16.089954 4431 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:53:16.090391 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:16.090042 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs podName:5a4476b6-e86f-49cd-a813-8cffdfcc3ca6 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:20.090030155 +0000 UTC m=+8.896551894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs") pod "network-metrics-daemon-8ntvq" (UID: "5a4476b6-e86f-49cd-a813-8cffdfcc3ca6") : secret "metrics-daemon-secret" not found Oct 30 05:53:16.090391 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.090238 4431 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:53:16.092267 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.092241 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15d1e924-1f5b-4867-aa83-cd43814eebbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rmk2m\" (UID: \"15d1e924-1f5b-4867-aa83-cd43814eebbb\") " pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:16.092430 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.092415 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c8f0b4-b6a7-4055-abca-b49923505435-metrics-tls\") pod \"dns-default-7b2h7\" (UID: \"f2c8f0b4-b6a7-4055-abca-b49923505435\") " pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:16.240411 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.240387 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rmk2m" Oct 30 05:53:16.240523 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.240464 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:16.559438 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.559418 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7b2h7"] Oct 30 05:53:16.560771 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:16.560744 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c8f0b4_b6a7_4055_abca_b49923505435.slice/crio-a82345b75da6c9f979bb14c468a8a34c60612fc087968a1a5c533bc7b33a7062 WatchSource:0}: Error finding container a82345b75da6c9f979bb14c468a8a34c60612fc087968a1a5c533bc7b33a7062: Status 404 returned error can't find the container with id a82345b75da6c9f979bb14c468a8a34c60612fc087968a1a5c533bc7b33a7062 Oct 30 05:53:16.562691 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.562675 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:53:16.570941 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:16.570920 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d1e924_1f5b_4867_aa83_cd43814eebbb.slice/crio-60cde44e6d829e0470e9a38281387a0ae536287f4815b57334f90fe1a75b35c2 WatchSource:0}: Error finding container 60cde44e6d829e0470e9a38281387a0ae536287f4815b57334f90fe1a75b35c2: Status 404 returned error can't find the container with id 60cde44e6d829e0470e9a38281387a0ae536287f4815b57334f90fe1a75b35c2 Oct 30 05:53:16.571834 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:16.571814 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rmk2m"] Oct 30 05:53:17.379226 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:17.379184 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7b2h7" event={"ID":"f2c8f0b4-b6a7-4055-abca-b49923505435","Type":"ContainerStarted","Data":"a82345b75da6c9f979bb14c468a8a34c60612fc087968a1a5c533bc7b33a7062"} Oct 30 05:53:17.381212 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:17.381178 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rmk2m" event={"ID":"15d1e924-1f5b-4867-aa83-cd43814eebbb","Type":"ContainerStarted","Data":"71e2afed8471c3273960ed6833b34a29754101df8f28424d1baab2f129128dd8"} Oct 30 05:53:17.381349 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:17.381239 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rmk2m" event={"ID":"15d1e924-1f5b-4867-aa83-cd43814eebbb","Type":"ContainerStarted","Data":"bea8e4dab14a82c659657d55f12806223013fdad0ddd196d4e586819537708d0"} Oct 30 05:53:17.381349 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:17.381269 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rmk2m" event={"ID":"15d1e924-1f5b-4867-aa83-cd43814eebbb","Type":"ContainerStarted","Data":"60cde44e6d829e0470e9a38281387a0ae536287f4815b57334f90fe1a75b35c2"} Oct 30 05:53:18.387103 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:18.387019 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7b2h7" event={"ID":"f2c8f0b4-b6a7-4055-abca-b49923505435","Type":"ContainerStarted","Data":"9704a52e47bf4d7d6ecf8e5265fb65581fce95b3937669e57d01e3c9bd2602cb"} Oct 30 05:53:18.387103 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:18.387093 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7b2h7" event={"ID":"f2c8f0b4-b6a7-4055-abca-b49923505435","Type":"ContainerStarted","Data":"892f1969764471fd44cdae78f4584a1d1da0466b372f83565a87829551eb412e"} Oct 30 05:53:18.387615 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:18.387150 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:18.802128 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:18.802098 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:19.391430 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:19.391392 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rmk2m" event={"ID":"15d1e924-1f5b-4867-aa83-cd43814eebbb","Type":"ContainerStarted","Data":"fe33068afae581544e235f6e44150c199244d576d2577df5c094fa67bc87a352"} Oct 30 05:53:20.115936 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:20.115884 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:20.118046 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:20.118019 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4476b6-e86f-49cd-a813-8cffdfcc3ca6-metrics-certs\") pod \"network-metrics-daemon-8ntvq\" (UID: \"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6\") " pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:20.139003 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:20.138972 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8ntvq" Oct 30 05:53:20.429351 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:20.429326 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8ntvq"] Oct 30 05:53:20.431659 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:20.431626 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4476b6_e86f_49cd_a813_8cffdfcc3ca6.slice/crio-76e6285b5ad7428e9e769b20db1853d82edf49a6d23fcd52408ad797e2438ec8 WatchSource:0}: Error finding container 76e6285b5ad7428e9e769b20db1853d82edf49a6d23fcd52408ad797e2438ec8: Status 404 returned error can't find the container with id 76e6285b5ad7428e9e769b20db1853d82edf49a6d23fcd52408ad797e2438ec8 Oct 30 05:53:21.399841 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:21.399809 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8ntvq" event={"ID":"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6","Type":"ContainerStarted","Data":"394ec9b685face434d1e000e98af5479dfce0f4dd1bb3b2575085b2c228cc7d0"} Oct 30 05:53:21.399961 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:21.399868 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8ntvq" event={"ID":"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6","Type":"ContainerStarted","Data":"4384398ea6874cf034962a50c45be0c07114d562886c34d0adff4917c01412eb"} Oct 30 05:53:21.399961 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:21.399892 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8ntvq" event={"ID":"5a4476b6-e86f-49cd-a813-8cffdfcc3ca6","Type":"ContainerStarted","Data":"76e6285b5ad7428e9e769b20db1853d82edf49a6d23fcd52408ad797e2438ec8"} Oct 30 05:53:22.815137 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.815104 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5"] Oct 30 05:53:22.817539 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.817520 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" Oct 30 05:53:22.851194 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.851164 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 05:53:22.851349 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.851333 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cg6pz\"" Oct 30 05:53:22.862843 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.862820 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5"] Oct 30 05:53:22.908827 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.908797 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-v8lfw"] Oct 30 05:53:22.910554 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.910535 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:22.914530 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.914511 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58bb7d9c8f-5xwq2"] Oct 30 05:53:22.916198 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.916182 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:22.922681 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.922664 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c755d5fbf-pcklc"] Oct 30 05:53:22.924416 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.924399 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:22.949631 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.949610 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba0e8d87-ccf1-4b09-8392-f9d8a02a9444-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9qgm5\" (UID: \"ba0e8d87-ccf1-4b09-8392-f9d8a02a9444\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" Oct 30 05:53:22.952439 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.952415 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 05:53:22.952525 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.952511 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 05:53:22.952667 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.952651 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 05:53:22.952925 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.952908 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-dgg8s\"" Oct 30 05:53:22.953003 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.952913 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-v8lfw"] Oct 30 05:53:22.964810 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.964789 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 05:53:22.965202 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.965161 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6k8z9\"" Oct 30 05:53:22.965330 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.965252 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 05:53:22.965330 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.965257 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 05:53:22.965432 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.965328 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 05:53:22.965432 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.965340 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 05:53:22.965432 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.965389 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 05:53:22.965432 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.965400 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 05:53:22.965599 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.965585 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 05:53:22.965903 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.965891 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58bb7d9c8f-5xwq2"] Oct 30 05:53:22.966195 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:22.966179 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c755d5fbf-pcklc"] Oct 30 05:53:23.040124 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.040100 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6ht9q\"" Oct 30 05:53:23.050703 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.050682 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/398c5a0b-aea9-499f-a423-6be32a46ee4d-registry-tls\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.050779 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.050708 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b306ac8e-f3f7-46f2-a46e-444351128037-tmp\") pod \"downloads-6dd789c8b7-v8lfw\" (UID: \"b306ac8e-f3f7-46f2-a46e-444351128037\") " pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:23.050779 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.050726 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/398c5a0b-aea9-499f-a423-6be32a46ee4d-trusted-ca\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.050779 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.050742 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/398c5a0b-aea9-499f-a423-6be32a46ee4d-bound-sa-token\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.050949 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.050816 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw52f\" (UniqueName: \"kubernetes.io/projected/c563b8c6-f410-400e-9031-f8e8c4e55bfd-kube-api-access-gw52f\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.050949 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.050876 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-serving-cert\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.050949 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.050916 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-oauth-config\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.051092 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.050976 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba0e8d87-ccf1-4b09-8392-f9d8a02a9444-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9qgm5\" (UID: \"ba0e8d87-ccf1-4b09-8392-f9d8a02a9444\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" Oct 30 05:53:23.051092 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051059 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-service-ca\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.051092 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051085 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ctt\" (UniqueName: \"kubernetes.io/projected/398c5a0b-aea9-499f-a423-6be32a46ee4d-kube-api-access-95ctt\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.051240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051101 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-config\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.051240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051126 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph822\" (UniqueName: \"kubernetes.io/projected/b306ac8e-f3f7-46f2-a46e-444351128037-kube-api-access-ph822\") pod \"downloads-6dd789c8b7-v8lfw\" (UID: \"b306ac8e-f3f7-46f2-a46e-444351128037\") " pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:23.051240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051191 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/398c5a0b-aea9-499f-a423-6be32a46ee4d-installation-pull-secrets\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.051240 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051215 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c563b8c6-f410-400e-9031-f8e8c4e55bfd-tmp\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.051454 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051267 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/398c5a0b-aea9-499f-a423-6be32a46ee4d-registry-certificates\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.051454 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051321 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/398c5a0b-aea9-499f-a423-6be32a46ee4d-image-registry-private-configuration\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.051454 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051391 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/398c5a0b-aea9-499f-a423-6be32a46ee4d-ca-trust-extracted\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.051454 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.051409 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-oauth-serving-cert\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.053377 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.053357 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba0e8d87-ccf1-4b09-8392-f9d8a02a9444-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9qgm5\" (UID: \"ba0e8d87-ccf1-4b09-8392-f9d8a02a9444\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" Oct 30 05:53:23.127647 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.127595 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" Oct 30 05:53:23.152689 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.152667 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw52f\" (UniqueName: \"kubernetes.io/projected/c563b8c6-f410-400e-9031-f8e8c4e55bfd-kube-api-access-gw52f\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.152783 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.152699 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-serving-cert\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.152783 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.152724 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-oauth-config\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.152783 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.152754 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-service-ca\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.152783 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.152774 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ctt\" (UniqueName: \"kubernetes.io/projected/398c5a0b-aea9-499f-a423-6be32a46ee4d-kube-api-access-95ctt\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.152956 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.152790 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-config\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.152956 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.152916 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph822\" (UniqueName: \"kubernetes.io/projected/b306ac8e-f3f7-46f2-a46e-444351128037-kube-api-access-ph822\") pod \"downloads-6dd789c8b7-v8lfw\" (UID: \"b306ac8e-f3f7-46f2-a46e-444351128037\") " pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:23.153047 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.152994 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/398c5a0b-aea9-499f-a423-6be32a46ee4d-installation-pull-secrets\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.153104 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153071 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c563b8c6-f410-400e-9031-f8e8c4e55bfd-tmp\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.153155 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153131 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/398c5a0b-aea9-499f-a423-6be32a46ee4d-registry-certificates\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153477 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c563b8c6-f410-400e-9031-f8e8c4e55bfd-tmp\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153528 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/398c5a0b-aea9-499f-a423-6be32a46ee4d-image-registry-private-configuration\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153592 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/398c5a0b-aea9-499f-a423-6be32a46ee4d-ca-trust-extracted\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153607 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-service-ca\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153619 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-oauth-serving-cert\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153645 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/398c5a0b-aea9-499f-a423-6be32a46ee4d-registry-tls\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153671 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b306ac8e-f3f7-46f2-a46e-444351128037-tmp\") pod \"downloads-6dd789c8b7-v8lfw\" (UID: \"b306ac8e-f3f7-46f2-a46e-444351128037\") " pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153693 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/398c5a0b-aea9-499f-a423-6be32a46ee4d-trusted-ca\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.153715 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/398c5a0b-aea9-499f-a423-6be32a46ee4d-bound-sa-token\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.154005 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-config\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.154330 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-oauth-serving-cert\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.154387 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.154347 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b306ac8e-f3f7-46f2-a46e-444351128037-tmp\") pod \"downloads-6dd789c8b7-v8lfw\" (UID: \"b306ac8e-f3f7-46f2-a46e-444351128037\") " pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:23.155337 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.154442 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/398c5a0b-aea9-499f-a423-6be32a46ee4d-registry-certificates\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.155337 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.154844 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/398c5a0b-aea9-499f-a423-6be32a46ee4d-trusted-ca\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.155337 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.154910 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/398c5a0b-aea9-499f-a423-6be32a46ee4d-ca-trust-extracted\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.156130 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.156096 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-serving-cert\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.156130 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.156124 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-oauth-config\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.156979 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.156954 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/398c5a0b-aea9-499f-a423-6be32a46ee4d-registry-tls\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.157189 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.157163 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/398c5a0b-aea9-499f-a423-6be32a46ee4d-image-registry-private-configuration\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.157372 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.157353 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/398c5a0b-aea9-499f-a423-6be32a46ee4d-installation-pull-secrets\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.180382 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.180362 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/398c5a0b-aea9-499f-a423-6be32a46ee4d-bound-sa-token\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.180663 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.180645 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph822\" (UniqueName: \"kubernetes.io/projected/b306ac8e-f3f7-46f2-a46e-444351128037-kube-api-access-ph822\") pod \"downloads-6dd789c8b7-v8lfw\" (UID: \"b306ac8e-f3f7-46f2-a46e-444351128037\") " pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:23.188795 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.188765 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw52f\" (UniqueName: \"kubernetes.io/projected/c563b8c6-f410-400e-9031-f8e8c4e55bfd-kube-api-access-gw52f\") pod \"console-c755d5fbf-pcklc\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.189749 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.189729 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ctt\" (UniqueName: \"kubernetes.io/projected/398c5a0b-aea9-499f-a423-6be32a46ee4d-kube-api-access-95ctt\") pod \"image-registry-58bb7d9c8f-5xwq2\" (UID: \"398c5a0b-aea9-499f-a423-6be32a46ee4d\") " pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.221570 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.221548 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:23.227196 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.227174 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:23.234011 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.233986 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:23.420888 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.420856 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5"] Oct 30 05:53:23.421881 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:23.421860 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0e8d87_ccf1_4b09_8392_f9d8a02a9444.slice/crio-467311a14bed330bbc8263dc44ba6acac6eb8cb72d5b55f326526d3a3e4e522a WatchSource:0}: Error finding container 467311a14bed330bbc8263dc44ba6acac6eb8cb72d5b55f326526d3a3e4e522a: Status 404 returned error can't find the container with id 467311a14bed330bbc8263dc44ba6acac6eb8cb72d5b55f326526d3a3e4e522a Oct 30 05:53:23.526153 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.526132 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-v8lfw"] Oct 30 05:53:23.527763 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:23.527731 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb306ac8e_f3f7_46f2_a46e_444351128037.slice/crio-f07ff9ff1295f7c5e1203987a5fba0968a78c90bd7e025246892cd1fd0f458b7 WatchSource:0}: Error finding container f07ff9ff1295f7c5e1203987a5fba0968a78c90bd7e025246892cd1fd0f458b7: Status 404 returned error can't find the container with id f07ff9ff1295f7c5e1203987a5fba0968a78c90bd7e025246892cd1fd0f458b7 Oct 30 05:53:23.534334 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.534312 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c755d5fbf-pcklc"] Oct 30 05:53:23.535501 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:23.535476 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc563b8c6_f410_400e_9031_f8e8c4e55bfd.slice/crio-37878af5231a1f179010bd724a51cf8fba2e227a500d29f8a302fc27916bf706 WatchSource:0}: Error finding container 37878af5231a1f179010bd724a51cf8fba2e227a500d29f8a302fc27916bf706: Status 404 returned error can't find the container with id 37878af5231a1f179010bd724a51cf8fba2e227a500d29f8a302fc27916bf706 Oct 30 05:53:23.539685 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:23.539666 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58bb7d9c8f-5xwq2"] Oct 30 05:53:23.540526 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:23.540505 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398c5a0b_aea9_499f_a423_6be32a46ee4d.slice/crio-56f929cd0379800f177e18a493a138cca69a14e314434792146dad27e376942d WatchSource:0}: Error finding container 56f929cd0379800f177e18a493a138cca69a14e314434792146dad27e376942d: Status 404 returned error can't find the container with id 56f929cd0379800f177e18a493a138cca69a14e314434792146dad27e376942d Oct 30 05:53:24.413065 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:24.412613 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" event={"ID":"398c5a0b-aea9-499f-a423-6be32a46ee4d","Type":"ContainerStarted","Data":"62207b711c181d021c34bcda5aeeef44b25db0f96e9c09d0f5a9412f346bc8a6"} Oct 30 05:53:24.413065 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:24.412672 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" event={"ID":"398c5a0b-aea9-499f-a423-6be32a46ee4d","Type":"ContainerStarted","Data":"56f929cd0379800f177e18a493a138cca69a14e314434792146dad27e376942d"} Oct 30 05:53:24.413065 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:24.412738 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:24.414893 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:24.414842 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-v8lfw" event={"ID":"b306ac8e-f3f7-46f2-a46e-444351128037","Type":"ContainerStarted","Data":"f07ff9ff1295f7c5e1203987a5fba0968a78c90bd7e025246892cd1fd0f458b7"} Oct 30 05:53:24.416491 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:24.416445 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" event={"ID":"ba0e8d87-ccf1-4b09-8392-f9d8a02a9444","Type":"ContainerStarted","Data":"467311a14bed330bbc8263dc44ba6acac6eb8cb72d5b55f326526d3a3e4e522a"} Oct 30 05:53:24.417797 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:24.417738 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c755d5fbf-pcklc" event={"ID":"c563b8c6-f410-400e-9031-f8e8c4e55bfd","Type":"ContainerStarted","Data":"37878af5231a1f179010bd724a51cf8fba2e227a500d29f8a302fc27916bf706"} Oct 30 05:53:25.073214 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:25.073179 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7b2h7_f2c8f0b4-b6a7-4055-abca-b49923505435/dns/0.log" Oct 30 05:53:25.217686 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:25.217656 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7b2h7_f2c8f0b4-b6a7-4055-abca-b49923505435/kube-rbac-proxy/0.log" Oct 30 05:53:25.416761 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:25.416675 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b4v22_099ea3b3-43c1-4bbf-81aa-b3cbd803df66/dns-node-resolver/0.log" Oct 30 05:53:25.423816 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:25.423789 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" event={"ID":"ba0e8d87-ccf1-4b09-8392-f9d8a02a9444","Type":"ContainerStarted","Data":"74956c2a8841cdd93ab62b6c6716b49c6849dba32b4dfd37049d74e948d1f132"} Oct 30 05:53:25.424104 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:25.424086 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" Oct 30 05:53:25.429918 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:25.429896 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" Oct 30 05:53:25.464957 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:25.464896 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9qgm5" podStartSLOduration=2.427481558 podStartE2EDuration="3.464879329s" totalImagesPullingTime="1.037397771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:25.46443048 +0000 UTC m=+14.270952228" watchObservedRunningTime="2026-10-30 05:53:25.464879329 +0000 UTC m=+14.271401081" Oct 30 05:53:25.465057 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:25.465006 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" podStartSLOduration=3.465001058 podStartE2EDuration="3.465001058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:24.458837416 +0000 UTC m=+13.265359162" watchObservedRunningTime="2026-10-30 05:53:25.465001058 +0000 UTC m=+14.271522815" Oct 30 05:53:26.017969 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.017936 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58bb7d9c8f-5xwq2_398c5a0b-aea9-499f-a423-6be32a46ee4d/registry/0.log" Oct 30 05:53:26.310776 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.310704 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z"] Oct 30 05:53:26.321822 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.321792 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.338490 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.338465 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z"] Oct 30 05:53:26.342590 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.342570 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 05:53:26.350528 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.349823 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 05:53:26.350528 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.350481 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:53:26.350528 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.350498 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:53:26.357901 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.357866 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:53:26.374557 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.374390 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6xk2w\"" Oct 30 05:53:26.481917 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.481877 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/808a129f-fd2d-475c-b70e-41bf5695490b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.482346 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.481936 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/808a129f-fd2d-475c-b70e-41bf5695490b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.482346 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.481984 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5nx\" (UniqueName: \"kubernetes.io/projected/808a129f-fd2d-475c-b70e-41bf5695490b-kube-api-access-fz5nx\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.482346 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.482030 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/808a129f-fd2d-475c-b70e-41bf5695490b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.582739 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.582710 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/808a129f-fd2d-475c-b70e-41bf5695490b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.582850 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.582785 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5nx\" (UniqueName: \"kubernetes.io/projected/808a129f-fd2d-475c-b70e-41bf5695490b-kube-api-access-fz5nx\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.582850 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.582833 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/808a129f-fd2d-475c-b70e-41bf5695490b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.582964 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.582884 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/808a129f-fd2d-475c-b70e-41bf5695490b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.583809 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.583723 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/808a129f-fd2d-475c-b70e-41bf5695490b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.585457 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.585436 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/808a129f-fd2d-475c-b70e-41bf5695490b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.585919 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.585891 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/808a129f-fd2d-475c-b70e-41bf5695490b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.607197 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.607173 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5nx\" (UniqueName: \"kubernetes.io/projected/808a129f-fd2d-475c-b70e-41bf5695490b-kube-api-access-fz5nx\") pod \"prometheus-operator-9cd89b74c-zzn7z\" (UID: \"808a129f-fd2d-475c-b70e-41bf5695490b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.636155 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.636132 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" Oct 30 05:53:26.816815 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.816742 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v2swz_a98d05d0-c37f-4a97-b943-fd4b728e29d0/node-ca/0.log" Oct 30 05:53:26.940444 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:26.940379 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z"] Oct 30 05:53:26.942796 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:26.942763 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808a129f_fd2d_475c_b70e_41bf5695490b.slice/crio-4ee3ed985c85f39b794777c8a59592cb11d0210714784162a575162058d251d1 WatchSource:0}: Error finding container 4ee3ed985c85f39b794777c8a59592cb11d0210714784162a575162058d251d1: Status 404 returned error can't find the container with id 4ee3ed985c85f39b794777c8a59592cb11d0210714784162a575162058d251d1 Oct 30 05:53:27.417289 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.417257 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jczv9_d6557aa0-6639-40cd-baed-ea6bf194348c/serve-healthcheck-canary/0.log" Oct 30 05:53:27.432656 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.432617 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c755d5fbf-pcklc" event={"ID":"c563b8c6-f410-400e-9031-f8e8c4e55bfd","Type":"ContainerStarted","Data":"815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff"} Oct 30 05:53:27.434854 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.434828 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" event={"ID":"808a129f-fd2d-475c-b70e-41bf5695490b","Type":"ContainerStarted","Data":"4ee3ed985c85f39b794777c8a59592cb11d0210714784162a575162058d251d1"} Oct 30 05:53:27.481649 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.481522 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c755d5fbf-pcklc" podStartSLOduration=2.455487235 podStartE2EDuration="5.481504525s" totalImagesPullingTime="3.02601729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:27.480725412 +0000 UTC m=+16.287247161" watchObservedRunningTime="2026-10-30 05:53:27.481504525 +0000 UTC m=+16.288026274" Oct 30 05:53:27.725998 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.725966 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9fdf99468-82gbx"] Oct 30 05:53:27.728274 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.728250 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.780702 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.780666 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 05:53:27.781325 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.781292 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9fdf99468-82gbx"] Oct 30 05:53:27.894386 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.894338 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1335b265-84d0-4776-b0a3-1f6c2caad9b5-tmp\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.894538 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.894403 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-oauth-serving-cert\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.894538 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.894451 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-serving-cert\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.894538 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.894479 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-oauth-config\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.894538 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.894510 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-service-ca\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.894678 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.894541 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-config\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.894678 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.894567 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-trusted-ca-bundle\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.894678 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.894647 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7jg\" (UniqueName: \"kubernetes.io/projected/1335b265-84d0-4776-b0a3-1f6c2caad9b5-kube-api-access-sm7jg\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.995578 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.995495 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-service-ca\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.995578 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.995536 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-config\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.995578 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.995571 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-trusted-ca-bundle\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.995821 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.995639 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7jg\" (UniqueName: \"kubernetes.io/projected/1335b265-84d0-4776-b0a3-1f6c2caad9b5-kube-api-access-sm7jg\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.995821 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.995674 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1335b265-84d0-4776-b0a3-1f6c2caad9b5-tmp\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.995821 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.995700 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-oauth-serving-cert\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.995821 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.995730 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-serving-cert\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.995821 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.995753 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-oauth-config\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.996667 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.996640 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1335b265-84d0-4776-b0a3-1f6c2caad9b5-tmp\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.997041 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.997012 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-oauth-serving-cert\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.997314 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.997270 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-trusted-ca-bundle\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.997401 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.997270 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-service-ca\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.997731 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.997707 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-config\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:27.999396 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:27.999376 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-oauth-config\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:28.001384 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.001361 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-serving-cert\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:28.019969 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.019946 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7jg\" (UniqueName: \"kubernetes.io/projected/1335b265-84d0-4776-b0a3-1f6c2caad9b5-kube-api-access-sm7jg\") pod \"console-9fdf99468-82gbx\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:28.041806 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.041784 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:28.354003 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.353979 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9fdf99468-82gbx"] Oct 30 05:53:28.355661 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:28.355628 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1335b265_84d0_4776_b0a3_1f6c2caad9b5.slice/crio-eac8b177319721a67209bea2a046c064d02645a928848fae4a5e14d789fe9f68 WatchSource:0}: Error finding container eac8b177319721a67209bea2a046c064d02645a928848fae4a5e14d789fe9f68: Status 404 returned error can't find the container with id eac8b177319721a67209bea2a046c064d02645a928848fae4a5e14d789fe9f68 Oct 30 05:53:28.394138 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.394110 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7b2h7" Oct 30 05:53:28.440056 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.439598 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fdf99468-82gbx" event={"ID":"1335b265-84d0-4776-b0a3-1f6c2caad9b5","Type":"ContainerStarted","Data":"12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd"} Oct 30 05:53:28.440056 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.439691 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fdf99468-82gbx" event={"ID":"1335b265-84d0-4776-b0a3-1f6c2caad9b5","Type":"ContainerStarted","Data":"eac8b177319721a67209bea2a046c064d02645a928848fae4a5e14d789fe9f68"} Oct 30 05:53:28.443660 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.443638 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" event={"ID":"808a129f-fd2d-475c-b70e-41bf5695490b","Type":"ContainerStarted","Data":"58b3eb4ac2210472faee797a3b437f03c0751a49aaccbe3ed0862ddf7126f59b"} Oct 30 05:53:28.443768 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.443681 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" event={"ID":"808a129f-fd2d-475c-b70e-41bf5695490b","Type":"ContainerStarted","Data":"9b30101c96bd36768563b0dd46b82715bd8ae6f52f25bcb4bf5eb99eb7b63617"} Oct 30 05:53:28.488349 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:28.488275 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9fdf99468-82gbx" podStartSLOduration=1.488257286 podStartE2EDuration="1.488257286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:28.487059305 +0000 UTC m=+17.293581066" watchObservedRunningTime="2026-10-30 05:53:28.488257286 +0000 UTC m=+17.294779036" Oct 30 05:53:30.011995 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:30.011966 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kf896" Oct 30 05:53:31.311067 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.311017 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zzn7z" podStartSLOduration=4.066648533 podStartE2EDuration="5.311003425s" totalImagesPullingTime="1.244354892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:28.528468561 +0000 UTC m=+17.334990311" watchObservedRunningTime="2026-10-30 05:53:31.311003425 +0000 UTC m=+20.117525172" Oct 30 05:53:31.311584 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.311185 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-92tw7"] Oct 30 05:53:31.316223 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.316196 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.322981 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.322949 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.323123 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.322995 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqcnn\" (UniqueName: \"kubernetes.io/projected/68e9dfda-59df-4877-976a-3f3ce8019df1-kube-api-access-lqcnn\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.323123 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.323105 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-tls\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.323245 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.323134 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-accelerators-collector-config\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.323245 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.323166 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-wtmp\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.323361 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.323288 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-textfile\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.323421 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.323360 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/68e9dfda-59df-4877-976a-3f3ce8019df1-root\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.323467 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.323425 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68e9dfda-59df-4877-976a-3f3ce8019df1-sys\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.323467 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.323456 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68e9dfda-59df-4877-976a-3f3ce8019df1-metrics-client-ca\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.367128 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.367106 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:53:31.367634 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.367612 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:53:31.367802 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.367777 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rkzps\"" Oct 30 05:53:31.382198 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.382171 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:53:31.424572 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.424547 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.424702 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.424575 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqcnn\" (UniqueName: \"kubernetes.io/projected/68e9dfda-59df-4877-976a-3f3ce8019df1-kube-api-access-lqcnn\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.424702 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.424609 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-tls\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.424702 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.424631 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-accelerators-collector-config\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.424702 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.424686 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-wtmp\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.424901 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.424717 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-textfile\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.424901 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.424753 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/68e9dfda-59df-4877-976a-3f3ce8019df1-root\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.424901 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.424783 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68e9dfda-59df-4877-976a-3f3ce8019df1-sys\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.424901 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.424805 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68e9dfda-59df-4877-976a-3f3ce8019df1-metrics-client-ca\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.426933 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.425263 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68e9dfda-59df-4877-976a-3f3ce8019df1-sys\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.426933 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.425275 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-wtmp\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.426933 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.425378 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/68e9dfda-59df-4877-976a-3f3ce8019df1-root\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.426933 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.425681 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-textfile\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.426933 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.425777 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68e9dfda-59df-4877-976a-3f3ce8019df1-metrics-client-ca\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.426933 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.425845 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-accelerators-collector-config\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.433942 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.433916 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.435750 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.435727 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/68e9dfda-59df-4877-976a-3f3ce8019df1-node-exporter-tls\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.450151 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.449784 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqcnn\" (UniqueName: \"kubernetes.io/projected/68e9dfda-59df-4877-976a-3f3ce8019df1-kube-api-access-lqcnn\") pod \"node-exporter-92tw7\" (UID: \"68e9dfda-59df-4877-976a-3f3ce8019df1\") " pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.635694 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:31.635619 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-92tw7" Oct 30 05:53:31.648862 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:31.648827 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e9dfda_59df_4877_976a_3f3ce8019df1.slice/crio-2ca08d8239ea2c79ece394da402a8d8672b35fc6e6cf101c8ac3b78f66eb271d WatchSource:0}: Error finding container 2ca08d8239ea2c79ece394da402a8d8672b35fc6e6cf101c8ac3b78f66eb271d: Status 404 returned error can't find the container with id 2ca08d8239ea2c79ece394da402a8d8672b35fc6e6cf101c8ac3b78f66eb271d Oct 30 05:53:32.457381 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:32.457349 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-92tw7" event={"ID":"68e9dfda-59df-4877-976a-3f3ce8019df1","Type":"ContainerStarted","Data":"2ca08d8239ea2c79ece394da402a8d8672b35fc6e6cf101c8ac3b78f66eb271d"} Oct 30 05:53:33.234978 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:33.234946 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:33.235150 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:33.235007 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:33.239580 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:33.239553 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:33.463082 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:33.463047 4431 generic.go:356] "Generic (PLEG): container finished" podID="68e9dfda-59df-4877-976a-3f3ce8019df1" containerID="88632f72b9e02122c9db63be6475e6da885b85216da79d0e712ecde124f63489" exitCode=0 Oct 30 05:53:33.463499 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:33.463139 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-92tw7" event={"ID":"68e9dfda-59df-4877-976a-3f3ce8019df1","Type":"ContainerDied","Data":"88632f72b9e02122c9db63be6475e6da885b85216da79d0e712ecde124f63489"} Oct 30 05:53:33.468081 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:33.468059 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:53:33.701811 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:53:33.701772 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:53:34.476931 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:34.476892 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-92tw7" event={"ID":"68e9dfda-59df-4877-976a-3f3ce8019df1","Type":"ContainerStarted","Data":"49474eaebd8bc52f0cc0ff327d286b191c453e92fabc7c35a4034bb9232d0a2a"} Oct 30 05:53:34.477423 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:34.476983 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-92tw7" event={"ID":"68e9dfda-59df-4877-976a-3f3ce8019df1","Type":"ContainerStarted","Data":"710d1d18afc57d8f894706dd6954ab908192fa45d52f5e4771c9ec400c1bd027"} Oct 30 05:53:38.042747 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:38.042708 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:38.043277 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:38.042768 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:38.048246 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:38.048206 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:38.108778 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:38.108723 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-92tw7" podStartSLOduration=5.951776591 podStartE2EDuration="7.108702872s" totalImagesPullingTime="1.077584588s" totalInitContainerRuntime="79.341693ms" isStatefulPod=true podCreationTimestamp="2026-10-30 05:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:34.550021752 +0000 UTC m=+23.356543500" watchObservedRunningTime="2026-10-30 05:53:38.108702872 +0000 UTC m=+26.915224619" Oct 30 05:53:38.194263 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:38.194228 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9fdf99468-82gbx"] Oct 30 05:53:38.497090 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:38.497061 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:53:40.504342 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:40.504286 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-v8lfw" event={"ID":"b306ac8e-f3f7-46f2-a46e-444351128037","Type":"ContainerStarted","Data":"2ab555ad1f1fb3098e3312067d8daeb41b40477ce7318e1e93c9e1f6d4d6edc0"} Oct 30 05:53:40.504820 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:40.504468 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:40.523544 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:40.523518 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-v8lfw" Oct 30 05:53:40.579969 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:40.579913 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-v8lfw" podStartSLOduration=2.042745043 podStartE2EDuration="18.579894219s" totalImagesPullingTime="16.537149176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:40.565810815 +0000 UTC m=+29.372332595" watchObservedRunningTime="2026-10-30 05:53:40.579894219 +0000 UTC m=+29.386415967" Oct 30 05:53:42.600839 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.600803 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59d49b5876-4b4bm"] Oct 30 05:53:42.629710 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.629677 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.638928 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.638902 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d49b5876-4b4bm"] Oct 30 05:53:42.731812 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.731780 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8626ac-c70f-4e72-968a-20484cf91b71-tmp\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.731996 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.731826 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-trusted-ca-bundle\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.731996 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.731851 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7hdh\" (UniqueName: \"kubernetes.io/projected/0e8626ac-c70f-4e72-968a-20484cf91b71-kube-api-access-w7hdh\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.731996 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.731928 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-oauth-serving-cert\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.732167 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.732039 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-service-ca\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.732167 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.732124 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-console-config\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.732167 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.732156 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-serving-cert\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.732333 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.732182 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-oauth-config\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.833541 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.833497 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-console-config\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.833709 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.833552 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-serving-cert\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.833709 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.833571 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-oauth-config\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.833709 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.833605 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8626ac-c70f-4e72-968a-20484cf91b71-tmp\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.833709 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.833634 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-trusted-ca-bundle\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.833709 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.833660 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7hdh\" (UniqueName: \"kubernetes.io/projected/0e8626ac-c70f-4e72-968a-20484cf91b71-kube-api-access-w7hdh\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.833709 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.833684 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-oauth-serving-cert\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.834052 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.833716 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-service-ca\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.834377 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.834349 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-console-config\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.834650 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.834630 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-service-ca\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.834725 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.834697 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8626ac-c70f-4e72-968a-20484cf91b71-tmp\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.834938 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.834915 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-oauth-serving-cert\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.835705 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.835681 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-trusted-ca-bundle\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.836441 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.836419 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-serving-cert\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.837889 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.837869 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-oauth-config\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:42.974229 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:42.974198 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7hdh\" (UniqueName: \"kubernetes.io/projected/0e8626ac-c70f-4e72-968a-20484cf91b71-kube-api-access-w7hdh\") pod \"console-59d49b5876-4b4bm\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:43.259003 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:43.258919 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:43.622200 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:43.621612 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d49b5876-4b4bm"] Oct 30 05:53:43.623949 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:53:43.623913 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8626ac_c70f_4e72_968a_20484cf91b71.slice/crio-f05fb5c65534587e2ee6c10128982d754ff1d1eaad2f2134c96dc8871955712b WatchSource:0}: Error finding container f05fb5c65534587e2ee6c10128982d754ff1d1eaad2f2134c96dc8871955712b: Status 404 returned error can't find the container with id f05fb5c65534587e2ee6c10128982d754ff1d1eaad2f2134c96dc8871955712b Oct 30 05:53:44.527163 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:44.527119 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d49b5876-4b4bm" event={"ID":"0e8626ac-c70f-4e72-968a-20484cf91b71","Type":"ContainerStarted","Data":"0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f"} Oct 30 05:53:44.527488 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:44.527189 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d49b5876-4b4bm" event={"ID":"0e8626ac-c70f-4e72-968a-20484cf91b71","Type":"ContainerStarted","Data":"f05fb5c65534587e2ee6c10128982d754ff1d1eaad2f2134c96dc8871955712b"} Oct 30 05:53:45.429860 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:45.429831 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58bb7d9c8f-5xwq2" Oct 30 05:53:45.522354 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:45.521472 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59d49b5876-4b4bm" podStartSLOduration=3.521448812 podStartE2EDuration="3.521448812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:44.606211916 +0000 UTC m=+33.412733664" watchObservedRunningTime="2026-10-30 05:53:45.521448812 +0000 UTC m=+34.327970562" Oct 30 05:53:48.804793 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:48.804762 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2jjjp" Oct 30 05:53:53.259659 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:53.259622 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:53.260139 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:53.259713 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:53.264036 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:53.264015 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:53.568722 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:53.568658 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:53:53.661438 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:53:53.661375 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c755d5fbf-pcklc"] Oct 30 05:54:03.699104 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:54:03.699059 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:54:04.520330 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.520265 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9fdf99468-82gbx" podUID="1335b265-84d0-4776-b0a3-1f6c2caad9b5" containerName="console" containerID="cri-o://12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd" gracePeriod=15 Oct 30 05:54:04.791690 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.791630 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:54:04.909668 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.909636 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-service-ca\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-service-ca\") pod \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " Oct 30 05:54:04.909802 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.909691 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1335b265-84d0-4776-b0a3-1f6c2caad9b5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1335b265-84d0-4776-b0a3-1f6c2caad9b5-tmp\") pod \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " Oct 30 05:54:04.909802 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.909715 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-config\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-config\") pod \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " Oct 30 05:54:04.909802 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.909765 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1335b265-84d0-4776-b0a3-1f6c2caad9b5-kube-api-access-sm7jg\" (UniqueName: \"kubernetes.io/projected/1335b265-84d0-4776-b0a3-1f6c2caad9b5-kube-api-access-sm7jg\") pod \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " Oct 30 05:54:04.909960 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.909798 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-oauth-serving-cert\") pod \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " Oct 30 05:54:04.909960 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.909845 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-oauth-config\") pod \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " Oct 30 05:54:04.909960 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.909880 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-trusted-ca-bundle\") pod \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " Oct 30 05:54:04.909960 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.909914 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-serving-cert\") pod \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\" (UID: \"1335b265-84d0-4776-b0a3-1f6c2caad9b5\") " Oct 30 05:54:04.910157 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.909964 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1335b265-84d0-4776-b0a3-1f6c2caad9b5-tmp" pod "1335b265-84d0-4776-b0a3-1f6c2caad9b5" (UID: "1335b265-84d0-4776-b0a3-1f6c2caad9b5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:54:04.910157 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.910118 4431 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1335b265-84d0-4776-b0a3-1f6c2caad9b5-tmp\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:04.910267 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.910178 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-service-ca" pod "1335b265-84d0-4776-b0a3-1f6c2caad9b5" (UID: "1335b265-84d0-4776-b0a3-1f6c2caad9b5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:04.910349 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.910323 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-oauth-serving-cert" pod "1335b265-84d0-4776-b0a3-1f6c2caad9b5" (UID: "1335b265-84d0-4776-b0a3-1f6c2caad9b5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:04.910413 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.910330 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-config" pod "1335b265-84d0-4776-b0a3-1f6c2caad9b5" (UID: "1335b265-84d0-4776-b0a3-1f6c2caad9b5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:04.910469 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.910417 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-trusted-ca-bundle" pod "1335b265-84d0-4776-b0a3-1f6c2caad9b5" (UID: "1335b265-84d0-4776-b0a3-1f6c2caad9b5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:04.911974 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.911944 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1335b265-84d0-4776-b0a3-1f6c2caad9b5-kube-api-access-sm7jg" pod "1335b265-84d0-4776-b0a3-1f6c2caad9b5" (UID: "1335b265-84d0-4776-b0a3-1f6c2caad9b5"). InnerVolumeSpecName "kube-api-access-sm7jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:04.911974 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.911960 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-oauth-config" pod "1335b265-84d0-4776-b0a3-1f6c2caad9b5" (UID: "1335b265-84d0-4776-b0a3-1f6c2caad9b5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:04.912123 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:04.912017 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-serving-cert" pod "1335b265-84d0-4776-b0a3-1f6c2caad9b5" (UID: "1335b265-84d0-4776-b0a3-1f6c2caad9b5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:05.010402 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.010378 4431 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-serving-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:05.010402 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.010401 4431 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:05.010555 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.010416 4431 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:05.010555 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.010430 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm7jg\" (UniqueName: \"kubernetes.io/projected/1335b265-84d0-4776-b0a3-1f6c2caad9b5-kube-api-access-sm7jg\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:05.010555 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.010444 4431 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-oauth-serving-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:05.010555 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.010457 4431 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1335b265-84d0-4776-b0a3-1f6c2caad9b5-console-oauth-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:05.010555 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.010470 4431 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1335b265-84d0-4776-b0a3-1f6c2caad9b5-trusted-ca-bundle\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:05.606007 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.605922 4431 generic.go:356] "Generic (PLEG): container finished" podID="1335b265-84d0-4776-b0a3-1f6c2caad9b5" containerID="12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd" exitCode=0 Oct 30 05:54:05.606007 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.605967 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fdf99468-82gbx" Oct 30 05:54:05.606007 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.605979 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fdf99468-82gbx" event={"ID":"1335b265-84d0-4776-b0a3-1f6c2caad9b5","Type":"ContainerDied","Data":"12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd"} Oct 30 05:54:05.606215 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.606016 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fdf99468-82gbx" event={"ID":"1335b265-84d0-4776-b0a3-1f6c2caad9b5","Type":"ContainerDied","Data":"eac8b177319721a67209bea2a046c064d02645a928848fae4a5e14d789fe9f68"} Oct 30 05:54:05.606215 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.606045 4431 scope.go:122] "RemoveContainer" containerID="12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd" Oct 30 05:54:05.615471 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.615448 4431 scope.go:122] "RemoveContainer" containerID="12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd" Oct 30 05:54:05.615749 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:54:05.615728 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd\": container with ID starting with 12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd not found: ID does not exist" containerID="12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd" Oct 30 05:54:05.615799 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.615758 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd"} err="failed to get container status \"12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd\": rpc error: code = NotFound desc = could not find container \"12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd\": container with ID starting with 12dd64453163981d8ef62d608411de68bf4626436b338e8ad7ebd96ff4bd3dbd not found: ID does not exist" Oct 30 05:54:05.658227 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.658200 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9fdf99468-82gbx"] Oct 30 05:54:05.669811 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:05.669786 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9fdf99468-82gbx"] Oct 30 05:54:07.334797 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:07.334764 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1335b265-84d0-4776-b0a3-1f6c2caad9b5" path="/var/lib/kubelet/pods/1335b265-84d0-4776-b0a3-1f6c2caad9b5/volumes" Oct 30 05:54:18.682025 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:18.681973 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c755d5fbf-pcklc" podUID="c563b8c6-f410-400e-9031-f8e8c4e55bfd" containerName="console" containerID="cri-o://815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff" gracePeriod=15 Oct 30 05:54:18.955320 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:18.955279 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:54:19.107246 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107215 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-oauth-config\") pod \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " Oct 30 05:54:19.107397 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107251 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-config\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-config\") pod \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " Oct 30 05:54:19.107397 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107281 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c563b8c6-f410-400e-9031-f8e8c4e55bfd-kube-api-access-gw52f\" (UniqueName: \"kubernetes.io/projected/c563b8c6-f410-400e-9031-f8e8c4e55bfd-kube-api-access-gw52f\") pod \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " Oct 30 05:54:19.107397 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107325 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c563b8c6-f410-400e-9031-f8e8c4e55bfd-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c563b8c6-f410-400e-9031-f8e8c4e55bfd-tmp\") pod \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " Oct 30 05:54:19.107397 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107385 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-service-ca\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-service-ca\") pod \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " Oct 30 05:54:19.107605 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107424 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-serving-cert\") pod \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " Oct 30 05:54:19.107605 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107448 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-oauth-serving-cert\") pod \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\" (UID: \"c563b8c6-f410-400e-9031-f8e8c4e55bfd\") " Oct 30 05:54:19.107737 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107711 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c563b8c6-f410-400e-9031-f8e8c4e55bfd-tmp" pod "c563b8c6-f410-400e-9031-f8e8c4e55bfd" (UID: "c563b8c6-f410-400e-9031-f8e8c4e55bfd"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:54:19.107801 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107771 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-config" pod "c563b8c6-f410-400e-9031-f8e8c4e55bfd" (UID: "c563b8c6-f410-400e-9031-f8e8c4e55bfd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:19.107863 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107808 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-service-ca" pod "c563b8c6-f410-400e-9031-f8e8c4e55bfd" (UID: "c563b8c6-f410-400e-9031-f8e8c4e55bfd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:19.107933 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.107914 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-oauth-serving-cert" pod "c563b8c6-f410-400e-9031-f8e8c4e55bfd" (UID: "c563b8c6-f410-400e-9031-f8e8c4e55bfd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:19.109476 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.109453 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c563b8c6-f410-400e-9031-f8e8c4e55bfd-kube-api-access-gw52f" pod "c563b8c6-f410-400e-9031-f8e8c4e55bfd" (UID: "c563b8c6-f410-400e-9031-f8e8c4e55bfd"). InnerVolumeSpecName "kube-api-access-gw52f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:19.109574 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.109494 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-oauth-config" pod "c563b8c6-f410-400e-9031-f8e8c4e55bfd" (UID: "c563b8c6-f410-400e-9031-f8e8c4e55bfd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:19.109574 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.109559 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-serving-cert" pod "c563b8c6-f410-400e-9031-f8e8c4e55bfd" (UID: "c563b8c6-f410-400e-9031-f8e8c4e55bfd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:19.161798 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.161772 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58849b4f7d-4clbp"] Oct 30 05:54:19.162016 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.162004 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1335b265-84d0-4776-b0a3-1f6c2caad9b5" containerName="console" Oct 30 05:54:19.162056 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.162017 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1335b265-84d0-4776-b0a3-1f6c2caad9b5" containerName="console" Oct 30 05:54:19.162056 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.162028 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c563b8c6-f410-400e-9031-f8e8c4e55bfd" containerName="console" Oct 30 05:54:19.162056 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.162033 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c563b8c6-f410-400e-9031-f8e8c4e55bfd" containerName="console" Oct 30 05:54:19.162142 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.162072 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1335b265-84d0-4776-b0a3-1f6c2caad9b5" containerName="console" Oct 30 05:54:19.162142 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.162081 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c563b8c6-f410-400e-9031-f8e8c4e55bfd" containerName="console" Oct 30 05:54:19.178576 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.178554 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.186683 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.186659 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58849b4f7d-4clbp"] Oct 30 05:54:19.208454 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.208279 4431 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:19.208675 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.208620 4431 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-serving-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:19.208675 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.208646 4431 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-oauth-serving-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:19.208986 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.208891 4431 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-oauth-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:19.208986 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.208935 4431 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c563b8c6-f410-400e-9031-f8e8c4e55bfd-console-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:19.208986 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.208949 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw52f\" (UniqueName: \"kubernetes.io/projected/c563b8c6-f410-400e-9031-f8e8c4e55bfd-kube-api-access-gw52f\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:19.208986 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.208962 4431 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c563b8c6-f410-400e-9031-f8e8c4e55bfd-tmp\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:19.309912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.309877 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4884t\" (UniqueName: \"kubernetes.io/projected/38186dd9-9779-4928-a9ae-f566d436bef5-kube-api-access-4884t\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.309912 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.309914 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-oauth-serving-cert\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.310120 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.309942 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38186dd9-9779-4928-a9ae-f566d436bef5-console-serving-cert\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.310120 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.309959 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38186dd9-9779-4928-a9ae-f566d436bef5-console-oauth-config\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.310120 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.310012 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-trusted-ca-bundle\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.310223 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.310132 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-console-config\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.310223 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.310155 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38186dd9-9779-4928-a9ae-f566d436bef5-tmp\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.310223 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.310185 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-service-ca\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.410825 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.410797 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4884t\" (UniqueName: \"kubernetes.io/projected/38186dd9-9779-4928-a9ae-f566d436bef5-kube-api-access-4884t\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.410972 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.410830 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-oauth-serving-cert\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.410972 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.410878 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38186dd9-9779-4928-a9ae-f566d436bef5-console-serving-cert\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.410972 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.410937 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38186dd9-9779-4928-a9ae-f566d436bef5-console-oauth-config\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.411142 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.410976 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-trusted-ca-bundle\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.411142 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.411008 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-console-config\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.411142 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.411033 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38186dd9-9779-4928-a9ae-f566d436bef5-tmp\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.411142 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.411109 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-service-ca\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.411565 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.411533 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38186dd9-9779-4928-a9ae-f566d436bef5-tmp\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.411898 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.411874 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-oauth-serving-cert\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.411987 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.411906 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-service-ca\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.412090 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.412067 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-trusted-ca-bundle\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.412333 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.412285 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38186dd9-9779-4928-a9ae-f566d436bef5-console-config\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.413511 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.413492 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38186dd9-9779-4928-a9ae-f566d436bef5-console-serving-cert\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.413591 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.413528 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38186dd9-9779-4928-a9ae-f566d436bef5-console-oauth-config\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.434195 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.434175 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4884t\" (UniqueName: \"kubernetes.io/projected/38186dd9-9779-4928-a9ae-f566d436bef5-kube-api-access-4884t\") pod \"console-58849b4f7d-4clbp\" (UID: \"38186dd9-9779-4928-a9ae-f566d436bef5\") " pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.507175 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.507127 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:19.653692 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.653665 4431 generic.go:356] "Generic (PLEG): container finished" podID="c563b8c6-f410-400e-9031-f8e8c4e55bfd" containerID="815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff" exitCode=0 Oct 30 05:54:19.653829 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.653711 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c755d5fbf-pcklc" Oct 30 05:54:19.653829 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.653729 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c755d5fbf-pcklc" event={"ID":"c563b8c6-f410-400e-9031-f8e8c4e55bfd","Type":"ContainerDied","Data":"815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff"} Oct 30 05:54:19.653829 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.653767 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c755d5fbf-pcklc" event={"ID":"c563b8c6-f410-400e-9031-f8e8c4e55bfd","Type":"ContainerDied","Data":"37878af5231a1f179010bd724a51cf8fba2e227a500d29f8a302fc27916bf706"} Oct 30 05:54:19.653829 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.653786 4431 scope.go:122] "RemoveContainer" containerID="815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff" Oct 30 05:54:19.661037 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.661020 4431 scope.go:122] "RemoveContainer" containerID="815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff" Oct 30 05:54:19.661339 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:54:19.661292 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff\": container with ID starting with 815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff not found: ID does not exist" containerID="815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff" Oct 30 05:54:19.661433 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.661346 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff"} err="failed to get container status \"815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff\": rpc error: code = NotFound desc = could not find container \"815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff\": container with ID starting with 815302f41a348ba903ca5aa80cd82f38c9a47047df99d16fe0ce3484435471ff not found: ID does not exist" Oct 30 05:54:19.704576 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.704552 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c755d5fbf-pcklc"] Oct 30 05:54:19.709141 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.709120 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c755d5fbf-pcklc"] Oct 30 05:54:19.820323 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:19.820185 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58849b4f7d-4clbp"] Oct 30 05:54:19.822926 ip-10-0-136-123 kubenswrapper[4431]: W1030 05:54:19.822904 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38186dd9_9779_4928_a9ae_f566d436bef5.slice/crio-cc48e6d425377155b697fa714890898e06f7dd834c5b7a499ab35c2e9ddd47a9 WatchSource:0}: Error finding container cc48e6d425377155b697fa714890898e06f7dd834c5b7a499ab35c2e9ddd47a9: Status 404 returned error can't find the container with id cc48e6d425377155b697fa714890898e06f7dd834c5b7a499ab35c2e9ddd47a9 Oct 30 05:54:20.659168 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:20.659132 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58849b4f7d-4clbp" event={"ID":"38186dd9-9779-4928-a9ae-f566d436bef5","Type":"ContainerStarted","Data":"ab0a4b2e54ffee6f31605df0d34cae39ec7fdbcde6653188a8fadd0303256197"} Oct 30 05:54:20.659391 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:20.659212 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58849b4f7d-4clbp" event={"ID":"38186dd9-9779-4928-a9ae-f566d436bef5","Type":"ContainerStarted","Data":"cc48e6d425377155b697fa714890898e06f7dd834c5b7a499ab35c2e9ddd47a9"} Oct 30 05:54:20.702204 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:20.702156 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58849b4f7d-4clbp" podStartSLOduration=1.7021428950000002 podStartE2EDuration="1.702142895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:20.701730747 +0000 UTC m=+69.508252495" watchObservedRunningTime="2026-10-30 05:54:20.702142895 +0000 UTC m=+69.508664643" Oct 30 05:54:21.334198 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:21.334167 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c563b8c6-f410-400e-9031-f8e8c4e55bfd" path="/var/lib/kubelet/pods/c563b8c6-f410-400e-9031-f8e8c4e55bfd/volumes" Oct 30 05:54:29.507746 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:29.507701 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:29.508346 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:29.507770 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:29.512907 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:29.512879 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:29.693056 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:29.693029 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58849b4f7d-4clbp" Oct 30 05:54:29.769030 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:29.768959 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d49b5876-4b4bm"] Oct 30 05:54:33.698455 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:54:33.698421 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:54:54.787576 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:54.787518 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59d49b5876-4b4bm" podUID="0e8626ac-c70f-4e72-968a-20484cf91b71" containerName="console" containerID="cri-o://0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f" gracePeriod=15 Oct 30 05:54:55.029235 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.029212 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:54:55.056538 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056489 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8626ac-c70f-4e72-968a-20484cf91b71-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8626ac-c70f-4e72-968a-20484cf91b71-tmp\") pod \"0e8626ac-c70f-4e72-968a-20484cf91b71\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " Oct 30 05:54:55.056538 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056528 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-trusted-ca-bundle\") pod \"0e8626ac-c70f-4e72-968a-20484cf91b71\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " Oct 30 05:54:55.056688 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056555 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8626ac-c70f-4e72-968a-20484cf91b71-kube-api-access-w7hdh\" (UniqueName: \"kubernetes.io/projected/0e8626ac-c70f-4e72-968a-20484cf91b71-kube-api-access-w7hdh\") pod \"0e8626ac-c70f-4e72-968a-20484cf91b71\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " Oct 30 05:54:55.056688 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056582 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-oauth-config\") pod \"0e8626ac-c70f-4e72-968a-20484cf91b71\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " Oct 30 05:54:55.056688 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056613 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-service-ca\") pod \"0e8626ac-c70f-4e72-968a-20484cf91b71\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " Oct 30 05:54:55.056688 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056643 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-oauth-serving-cert\") pod \"0e8626ac-c70f-4e72-968a-20484cf91b71\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " Oct 30 05:54:55.056893 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056749 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-serving-cert\") pod \"0e8626ac-c70f-4e72-968a-20484cf91b71\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " Oct 30 05:54:55.056893 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056781 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8626ac-c70f-4e72-968a-20484cf91b71-tmp" pod "0e8626ac-c70f-4e72-968a-20484cf91b71" (UID: "0e8626ac-c70f-4e72-968a-20484cf91b71"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:54:55.056893 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056798 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-console-config\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-console-config\") pod \"0e8626ac-c70f-4e72-968a-20484cf91b71\" (UID: \"0e8626ac-c70f-4e72-968a-20484cf91b71\") " Oct 30 05:54:55.057057 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.056934 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-trusted-ca-bundle" pod "0e8626ac-c70f-4e72-968a-20484cf91b71" (UID: "0e8626ac-c70f-4e72-968a-20484cf91b71"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:55.057165 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.057145 4431 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e8626ac-c70f-4e72-968a-20484cf91b71-tmp\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:55.057228 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.057174 4431 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-trusted-ca-bundle\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:55.057228 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.057141 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-oauth-serving-cert" pod "0e8626ac-c70f-4e72-968a-20484cf91b71" (UID: "0e8626ac-c70f-4e72-968a-20484cf91b71"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:55.057228 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.057167 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-service-ca" pod "0e8626ac-c70f-4e72-968a-20484cf91b71" (UID: "0e8626ac-c70f-4e72-968a-20484cf91b71"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:55.057405 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.057275 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-console-config" pod "0e8626ac-c70f-4e72-968a-20484cf91b71" (UID: "0e8626ac-c70f-4e72-968a-20484cf91b71"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:55.058694 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.058671 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-oauth-config" pod "0e8626ac-c70f-4e72-968a-20484cf91b71" (UID: "0e8626ac-c70f-4e72-968a-20484cf91b71"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:55.058791 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.058717 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-serving-cert" pod "0e8626ac-c70f-4e72-968a-20484cf91b71" (UID: "0e8626ac-c70f-4e72-968a-20484cf91b71"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:55.058837 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.058786 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8626ac-c70f-4e72-968a-20484cf91b71-kube-api-access-w7hdh" pod "0e8626ac-c70f-4e72-968a-20484cf91b71" (UID: "0e8626ac-c70f-4e72-968a-20484cf91b71"). InnerVolumeSpecName "kube-api-access-w7hdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:55.158533 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.158503 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7hdh\" (UniqueName: \"kubernetes.io/projected/0e8626ac-c70f-4e72-968a-20484cf91b71-kube-api-access-w7hdh\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:55.158533 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.158529 4431 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-oauth-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:55.158665 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.158539 4431 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:55.158665 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.158548 4431 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-oauth-serving-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:55.158665 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.158557 4431 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e8626ac-c70f-4e72-968a-20484cf91b71-console-serving-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:55.158665 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.158565 4431 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e8626ac-c70f-4e72-968a-20484cf91b71-console-config\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 05:54:55.768939 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.768904 4431 generic.go:356] "Generic (PLEG): container finished" podID="0e8626ac-c70f-4e72-968a-20484cf91b71" containerID="0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f" exitCode=0 Oct 30 05:54:55.769097 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.768953 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d49b5876-4b4bm" Oct 30 05:54:55.769097 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.768998 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d49b5876-4b4bm" event={"ID":"0e8626ac-c70f-4e72-968a-20484cf91b71","Type":"ContainerDied","Data":"0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f"} Oct 30 05:54:55.769097 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.769039 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d49b5876-4b4bm" event={"ID":"0e8626ac-c70f-4e72-968a-20484cf91b71","Type":"ContainerDied","Data":"f05fb5c65534587e2ee6c10128982d754ff1d1eaad2f2134c96dc8871955712b"} Oct 30 05:54:55.769097 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.769060 4431 scope.go:122] "RemoveContainer" containerID="0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f" Oct 30 05:54:55.776693 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.776677 4431 scope.go:122] "RemoveContainer" containerID="0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f" Oct 30 05:54:55.776939 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:54:55.776921 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f\": container with ID starting with 0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f not found: ID does not exist" containerID="0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f" Oct 30 05:54:55.776991 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.776946 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f"} err="failed to get container status \"0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f\": rpc error: code = NotFound desc = could not find container \"0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f\": container with ID starting with 0fb7d4fb62e7a6672d81ef226a00859cf2ee0093cdef87a49a5a442c06fc4a2f not found: ID does not exist" Oct 30 05:54:55.818870 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.818847 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d49b5876-4b4bm"] Oct 30 05:54:55.831158 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:55.831133 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59d49b5876-4b4bm"] Oct 30 05:54:57.334701 ip-10-0-136-123 kubenswrapper[4431]: I1030 05:54:57.334672 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8626ac-c70f-4e72-968a-20484cf91b71" path="/var/lib/kubelet/pods/0e8626ac-c70f-4e72-968a-20484cf91b71/volumes" Oct 30 05:55:03.699038 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:55:03.699001 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:55:33.699248 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:55:33.699168 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:56:03.698622 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:56:03.698581 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:56:33.699115 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:56:33.699076 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:57:03.698825 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:57:03.698748 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:57:33.699833 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:57:33.699787 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:58:03.698615 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:58:03.698579 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:58:33.698998 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:58:33.698956 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:59:03.699835 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:59:03.699793 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 05:59:33.698320 ip-10-0-136-123 kubenswrapper[4431]: E1030 05:59:33.698276 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:00:03.699560 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:00:03.699482 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:00:05.268427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.268391 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv"] Oct 30 06:00:05.268834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.268685 4431 cpu_manager.go:414] "containerMap: removing container" podUID="0e8626ac-c70f-4e72-968a-20484cf91b71" containerName="console" Oct 30 06:00:05.268834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.268702 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8626ac-c70f-4e72-968a-20484cf91b71" containerName="console" Oct 30 06:00:05.268834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.268776 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8626ac-c70f-4e72-968a-20484cf91b71" containerName="console" Oct 30 06:00:05.270747 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.270722 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.271672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.271647 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv"] Oct 30 06:00:05.273715 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.273695 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.287428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.287410 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 06:00:05.287510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.287475 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 06:00:05.287510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.287502 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-227tz\"" Oct 30 06:00:05.297628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.297607 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-rmcn5\"" Oct 30 06:00:05.306205 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.306181 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.306352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.306232 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.306352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.306293 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27wn\" (UniqueName: \"kubernetes.io/projected/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-kube-api-access-x27wn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.306453 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.306377 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.306453 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.306410 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8cfc\" (UniqueName: \"kubernetes.io/projected/b05e8390-21af-4168-8296-017c308398ff-kube-api-access-x8cfc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.306453 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.306442 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.307375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.307356 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv"] Oct 30 06:00:05.313114 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.313095 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv"] Oct 30 06:00:05.369959 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.369934 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr"] Oct 30 06:00:05.371849 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.371833 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.395847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.395826 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 06:00:05.395955 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.395854 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 06:00:05.396148 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.396133 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 06:00:05.396205 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.396199 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 06:00:05.396255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.396221 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 06:00:05.396383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.396371 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 06:00:05.396423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.396396 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dbg4t\"" Oct 30 06:00:05.403238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.403211 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr"] Oct 30 06:00:05.407629 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.407606 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.407726 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.407655 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27wn\" (UniqueName: \"kubernetes.io/projected/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-kube-api-access-x27wn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.407726 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.407688 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.407809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.407734 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.407878 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.407855 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0d0110-19d2-428c-8234-dd9269d82d1e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.407934 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.407900 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.408019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.407990 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8cfc\" (UniqueName: \"kubernetes.io/projected/b05e8390-21af-4168-8296-017c308398ff-kube-api-access-x8cfc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.408073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.408031 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.408073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.407996 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.408153 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.408093 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nbjh\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-kube-api-access-2nbjh\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.408153 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.408125 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.408228 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.408155 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.408228 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.408179 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.408335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.408287 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.408395 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.408342 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.408494 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.408469 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.408584 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.408568 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.431514 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.431487 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27wn\" (UniqueName: \"kubernetes.io/projected/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-kube-api-access-x27wn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.431802 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.431787 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8cfc\" (UniqueName: \"kubernetes.io/projected/b05e8390-21af-4168-8296-017c308398ff-kube-api-access-x8cfc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.509550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.509520 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.509684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.509583 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0d0110-19d2-428c-8234-dd9269d82d1e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.509684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.509616 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.509684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.509648 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.509684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.509670 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nbjh\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-kube-api-access-2nbjh\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.509897 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.509704 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.509897 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.509738 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.510329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.510277 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.511800 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.511776 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0d0110-19d2-428c-8234-dd9269d82d1e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.512014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.511993 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.512247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.512232 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.512329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.512292 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.533510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.533459 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nbjh\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-kube-api-access-2nbjh\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.533510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.533479 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-js6cr\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.583938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.583915 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:05.589410 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.589390 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:05.681665 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.681637 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:05.885082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.885015 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv"] Oct 30 06:00:05.888106 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:00:05.888067 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05e8390_21af_4168_8296_017c308398ff.slice/crio-f933149782c41e26f31f82f85c8e78965329d9e3bcf42c13acdf5f110b303b62 WatchSource:0}: Error finding container f933149782c41e26f31f82f85c8e78965329d9e3bcf42c13acdf5f110b303b62: Status 404 returned error can't find the container with id f933149782c41e26f31f82f85c8e78965329d9e3bcf42c13acdf5f110b303b62 Oct 30 06:00:05.889980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.889961 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:00:05.898389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.898369 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv"] Oct 30 06:00:05.900451 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:00:05.900431 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2853bfa8_bc18_470e_9f9b_3ccb36cdc9c4.slice/crio-af025cc5dfebf1874a8212e1ca0772eea866eaa562c81a9e13d870c93959e4e1 WatchSource:0}: Error finding container af025cc5dfebf1874a8212e1ca0772eea866eaa562c81a9e13d870c93959e4e1: Status 404 returned error can't find the container with id af025cc5dfebf1874a8212e1ca0772eea866eaa562c81a9e13d870c93959e4e1 Oct 30 06:00:05.983903 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:05.983884 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr"] Oct 30 06:00:06.718528 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:06.718493 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" event={"ID":"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4","Type":"ContainerStarted","Data":"af025cc5dfebf1874a8212e1ca0772eea866eaa562c81a9e13d870c93959e4e1"} Oct 30 06:00:06.720275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:06.720245 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" event={"ID":"ea0d0110-19d2-428c-8234-dd9269d82d1e","Type":"ContainerStarted","Data":"cb0346a6d5924836a3d661dd68ba250cf1e791c07e5354cff2e9de8b48734172"} Oct 30 06:00:06.721475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:06.721446 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" event={"ID":"b05e8390-21af-4168-8296-017c308398ff","Type":"ContainerStarted","Data":"f933149782c41e26f31f82f85c8e78965329d9e3bcf42c13acdf5f110b303b62"} Oct 30 06:00:12.747662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:12.747628 4431 generic.go:356] "Generic (PLEG): container finished" podID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerID="02dd69a13d9d7bea2b0a9388105402f889f754291935070fffa101317463915f" exitCode=0 Oct 30 06:00:12.748134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:12.747719 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" event={"ID":"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4","Type":"ContainerDied","Data":"02dd69a13d9d7bea2b0a9388105402f889f754291935070fffa101317463915f"} Oct 30 06:00:12.749286 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:12.749260 4431 generic.go:356] "Generic (PLEG): container finished" podID="b05e8390-21af-4168-8296-017c308398ff" containerID="a1e4bd9d91da378d15d2205d70913f604faf478dc9c81fdfb8c902cea76068db" exitCode=0 Oct 30 06:00:12.749415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:12.749352 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" event={"ID":"b05e8390-21af-4168-8296-017c308398ff","Type":"ContainerDied","Data":"a1e4bd9d91da378d15d2205d70913f604faf478dc9c81fdfb8c902cea76068db"} Oct 30 06:00:15.763277 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:15.763247 4431 generic.go:356] "Generic (PLEG): container finished" podID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerID="cc55f93405ef5a3df6c531bf11aea2bb17f3cf87ecb9cba00279df8f58acf490" exitCode=0 Oct 30 06:00:15.763755 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:15.763340 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" event={"ID":"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4","Type":"ContainerDied","Data":"cc55f93405ef5a3df6c531bf11aea2bb17f3cf87ecb9cba00279df8f58acf490"} Oct 30 06:00:15.765212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:15.765149 4431 generic.go:356] "Generic (PLEG): container finished" podID="b05e8390-21af-4168-8296-017c308398ff" containerID="09e5930ab20562938ac304db5d3bb18ec8ad4d7ee366f9dd03d0e23791e9bae8" exitCode=0 Oct 30 06:00:15.765280 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:15.765216 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" event={"ID":"b05e8390-21af-4168-8296-017c308398ff","Type":"ContainerDied","Data":"09e5930ab20562938ac304db5d3bb18ec8ad4d7ee366f9dd03d0e23791e9bae8"} Oct 30 06:00:21.820036 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:21.819998 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:00:21.820377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:21.820078 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:00:22.796325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:22.796281 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" event={"ID":"ea0d0110-19d2-428c-8234-dd9269d82d1e","Type":"ContainerStarted","Data":"bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276"} Oct 30 06:00:22.796590 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:22.796564 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:22.798134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:22.798109 4431 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-js6cr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:00:22.798227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:22.798166 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" podUID="ea0d0110-19d2-428c-8234-dd9269d82d1e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:00:22.870584 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:22.868447 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" podStartSLOduration=2.03941348 podStartE2EDuration="17.868415671s" totalImagesPullingTime="15.829002191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:22.861318567 +0000 UTC m=+431.667840315" watchObservedRunningTime="2026-10-30 06:00:22.868415671 +0000 UTC m=+431.674937407" Oct 30 06:00:23.802026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:23.801992 4431 generic.go:356] "Generic (PLEG): container finished" podID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerID="ba1ab99a92a86617dcb45ab3f825b194587a78a1304e517389647e773fa9e6d0" exitCode=0 Oct 30 06:00:23.802186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:23.802051 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" event={"ID":"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4","Type":"ContainerDied","Data":"ba1ab99a92a86617dcb45ab3f825b194587a78a1304e517389647e773fa9e6d0"} Oct 30 06:00:23.804094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:23.804072 4431 generic.go:356] "Generic (PLEG): container finished" podID="b05e8390-21af-4168-8296-017c308398ff" containerID="768994aa37c0cce5fcc428d187fce7fe9932eb5f47606fdee20454c921d6d67c" exitCode=0 Oct 30 06:00:23.804224 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:23.804153 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" event={"ID":"b05e8390-21af-4168-8296-017c308398ff","Type":"ContainerDied","Data":"768994aa37c0cce5fcc428d187fce7fe9932eb5f47606fdee20454c921d6d67c"} Oct 30 06:00:23.804770 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:23.804754 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:00:24.926269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:24.926244 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:24.951487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:24.951456 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:24.962794 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:24.962775 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-kube-api-access-x27wn\" (UniqueName: \"kubernetes.io/projected/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-kube-api-access-x27wn\") pod \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " Oct 30 06:00:24.962908 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:24.962825 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-util\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-util\") pod \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " Oct 30 06:00:24.962908 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:24.962885 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-bundle\") pod \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\" (UID: \"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4\") " Oct 30 06:00:24.963246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:24.963225 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-bundle" pod "2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" (UID: "2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:24.964865 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:24.964840 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-kube-api-access-x27wn" pod "2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" (UID: "2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4"). InnerVolumeSpecName "kube-api-access-x27wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:24.967132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:24.967109 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-util" pod "2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" (UID: "2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:25.064121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.064042 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-bundle\") pod \"b05e8390-21af-4168-8296-017c308398ff\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " Oct 30 06:00:25.064121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.064080 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-util\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-util\") pod \"b05e8390-21af-4168-8296-017c308398ff\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " Oct 30 06:00:25.064347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.064130 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05e8390-21af-4168-8296-017c308398ff-kube-api-access-x8cfc\" (UniqueName: \"kubernetes.io/projected/b05e8390-21af-4168-8296-017c308398ff-kube-api-access-x8cfc\") pod \"b05e8390-21af-4168-8296-017c308398ff\" (UID: \"b05e8390-21af-4168-8296-017c308398ff\") " Oct 30 06:00:25.064347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.064250 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27wn\" (UniqueName: \"kubernetes.io/projected/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-kube-api-access-x27wn\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:00:25.064347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.064261 4431 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-util\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:00:25.064347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.064270 4431 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4-bundle\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:00:25.064684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.064650 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-bundle" pod "b05e8390-21af-4168-8296-017c308398ff" (UID: "b05e8390-21af-4168-8296-017c308398ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:25.066185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.066165 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05e8390-21af-4168-8296-017c308398ff-kube-api-access-x8cfc" pod "b05e8390-21af-4168-8296-017c308398ff" (UID: "b05e8390-21af-4168-8296-017c308398ff"). InnerVolumeSpecName "kube-api-access-x8cfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:25.068191 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.068168 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-util" pod "b05e8390-21af-4168-8296-017c308398ff" (UID: "b05e8390-21af-4168-8296-017c308398ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:25.164563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.164543 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8cfc\" (UniqueName: \"kubernetes.io/projected/b05e8390-21af-4168-8296-017c308398ff-kube-api-access-x8cfc\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:00:25.164563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.164564 4431 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-bundle\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:00:25.164681 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.164573 4431 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b05e8390-21af-4168-8296-017c308398ff-util\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:00:25.812877 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.812846 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" Oct 30 06:00:25.813037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.812849 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445fnpv" event={"ID":"2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4","Type":"ContainerDied","Data":"af025cc5dfebf1874a8212e1ca0772eea866eaa562c81a9e13d870c93959e4e1"} Oct 30 06:00:25.813037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.812972 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af025cc5dfebf1874a8212e1ca0772eea866eaa562c81a9e13d870c93959e4e1" Oct 30 06:00:25.814925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.814881 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" event={"ID":"b05e8390-21af-4168-8296-017c308398ff","Type":"ContainerDied","Data":"f933149782c41e26f31f82f85c8e78965329d9e3bcf42c13acdf5f110b303b62"} Oct 30 06:00:25.814925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.814919 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpfnvv" Oct 30 06:00:25.815172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:25.814924 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f933149782c41e26f31f82f85c8e78965329d9e3bcf42c13acdf5f110b303b62" Oct 30 06:00:33.699017 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:00:33.698982 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:00:41.124593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124558 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6"] Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124807 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b05e8390-21af-4168-8296-017c308398ff" containerName="util" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124817 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05e8390-21af-4168-8296-017c308398ff" containerName="util" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124828 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b05e8390-21af-4168-8296-017c308398ff" containerName="extract" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124833 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05e8390-21af-4168-8296-017c308398ff" containerName="extract" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124841 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerName="util" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124846 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerName="util" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124853 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b05e8390-21af-4168-8296-017c308398ff" containerName="pull" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124857 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05e8390-21af-4168-8296-017c308398ff" containerName="pull" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124863 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerName="pull" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124868 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerName="pull" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124877 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerName="extract" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124882 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerName="extract" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124921 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2853bfa8-bc18-470e-9f9b-3ccb36cdc9c4" containerName="extract" Oct 30 06:00:41.125002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.124928 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05e8390-21af-4168-8296-017c308398ff" containerName="extract" Oct 30 06:00:41.128108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.128083 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.160766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.160735 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6"] Oct 30 06:00:41.185165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.185141 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 06:00:41.185165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.185161 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 06:00:41.185459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.185439 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 06:00:41.185584 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.185493 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 06:00:41.185584 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.185571 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n5fhg\"" Oct 30 06:00:41.185670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.185657 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 06:00:41.277318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.277275 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/26fb53d8-6a42-4581-a859-cd3f4e162629-manifests\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.277429 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.277319 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26fb53d8-6a42-4581-a859-cd3f4e162629-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.277429 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.277341 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26fb53d8-6a42-4581-a859-cd3f4e162629-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.277513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.277451 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/26fb53d8-6a42-4581-a859-cd3f4e162629-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.277513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.277478 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9sf\" (UniqueName: \"kubernetes.io/projected/26fb53d8-6a42-4581-a859-cd3f4e162629-kube-api-access-qx9sf\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.277513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.277494 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26fb53d8-6a42-4581-a859-cd3f4e162629-tmp\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.378488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.378413 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/26fb53d8-6a42-4581-a859-cd3f4e162629-manifests\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.378488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.378446 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26fb53d8-6a42-4581-a859-cd3f4e162629-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.378488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.378475 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26fb53d8-6a42-4581-a859-cd3f4e162629-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.378745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.378531 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/26fb53d8-6a42-4581-a859-cd3f4e162629-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.378745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.378693 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9sf\" (UniqueName: \"kubernetes.io/projected/26fb53d8-6a42-4581-a859-cd3f4e162629-kube-api-access-qx9sf\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.378745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.378739 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26fb53d8-6a42-4581-a859-cd3f4e162629-tmp\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.378901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.378847 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/26fb53d8-6a42-4581-a859-cd3f4e162629-manifests\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.380851 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.380827 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26fb53d8-6a42-4581-a859-cd3f4e162629-tmp\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.380978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.380890 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26fb53d8-6a42-4581-a859-cd3f4e162629-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.381288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.381269 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/26fb53d8-6a42-4581-a859-cd3f4e162629-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.381375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.381325 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26fb53d8-6a42-4581-a859-cd3f4e162629-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.409256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.409231 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9sf\" (UniqueName: \"kubernetes.io/projected/26fb53d8-6a42-4581-a859-cd3f4e162629-kube-api-access-qx9sf\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-mxrj6\" (UID: \"26fb53d8-6a42-4581-a859-cd3f4e162629\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.443143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.443123 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:41.885822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:41.885772 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6"] Oct 30 06:00:41.888354 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:00:41.888325 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fb53d8_6a42_4581_a859_cd3f4e162629.slice/crio-9377d76eb71032a8dbcebfb7dd6347919b891d3e4ae766c6a23854fc0d0b6e38 WatchSource:0}: Error finding container 9377d76eb71032a8dbcebfb7dd6347919b891d3e4ae766c6a23854fc0d0b6e38: Status 404 returned error can't find the container with id 9377d76eb71032a8dbcebfb7dd6347919b891d3e4ae766c6a23854fc0d0b6e38 Oct 30 06:00:42.873415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:42.873374 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" event={"ID":"26fb53d8-6a42-4581-a859-cd3f4e162629","Type":"ContainerStarted","Data":"9377d76eb71032a8dbcebfb7dd6347919b891d3e4ae766c6a23854fc0d0b6e38"} Oct 30 06:00:44.883052 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:44.882984 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" event={"ID":"26fb53d8-6a42-4581-a859-cd3f4e162629","Type":"ContainerStarted","Data":"8f68244f4b345a13926895bd68ef84857fc6004982b2104edd40ebf4dfcdd251"} Oct 30 06:00:45.892631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:45.892584 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" event={"ID":"26fb53d8-6a42-4581-a859-cd3f4e162629","Type":"ContainerDied","Data":"8f68244f4b345a13926895bd68ef84857fc6004982b2104edd40ebf4dfcdd251"} Oct 30 06:00:45.893028 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:45.892621 4431 generic.go:356] "Generic (PLEG): container finished" podID="26fb53d8-6a42-4581-a859-cd3f4e162629" containerID="8f68244f4b345a13926895bd68ef84857fc6004982b2104edd40ebf4dfcdd251" exitCode=0 Oct 30 06:00:46.898604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:46.898572 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" event={"ID":"26fb53d8-6a42-4581-a859-cd3f4e162629","Type":"ContainerStarted","Data":"5b4b7990bb8ad961615a7b76d5cfc81b63d7994e5ff3ab0db5d969301841e323"} Oct 30 06:00:46.898997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:46.898647 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:00:47.005060 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:47.005007 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" podStartSLOduration=2.353181475 podStartE2EDuration="6.004994288s" totalImagesPullingTime="3.179502869s" totalInitContainerRuntime="472.309944ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:46.992479861 +0000 UTC m=+455.799001609" watchObservedRunningTime="2026-10-30 06:00:47.004994288 +0000 UTC m=+455.811516036" Oct 30 06:00:57.905401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:00:57.905371 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-mxrj6" Oct 30 06:01:03.699369 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:01:03.699334 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:01:12.891136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:12.891097 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp"] Oct 30 06:01:12.894140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:12.894116 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:12.910992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:12.910965 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nr94k\"" Oct 30 06:01:12.911082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:12.910971 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 06:01:12.911082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:12.911026 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 06:01:12.918988 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:12.918966 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp"] Oct 30 06:01:13.007932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.007894 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a-tmp\") pod \"kube-auth-proxy-5c4bb65c68-jwxlp\" (UID: \"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.008072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.007978 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqblf\" (UniqueName: \"kubernetes.io/projected/d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a-kube-api-access-wqblf\") pod \"kube-auth-proxy-5c4bb65c68-jwxlp\" (UID: \"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.008072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.008019 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a-tls-certs\") pod \"kube-auth-proxy-5c4bb65c68-jwxlp\" (UID: \"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.109165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.109131 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a-tls-certs\") pod \"kube-auth-proxy-5c4bb65c68-jwxlp\" (UID: \"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.109326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.109186 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a-tmp\") pod \"kube-auth-proxy-5c4bb65c68-jwxlp\" (UID: \"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.109326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.109255 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqblf\" (UniqueName: \"kubernetes.io/projected/d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a-kube-api-access-wqblf\") pod \"kube-auth-proxy-5c4bb65c68-jwxlp\" (UID: \"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.111556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.111532 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a-tmp\") pod \"kube-auth-proxy-5c4bb65c68-jwxlp\" (UID: \"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.111756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.111735 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a-tls-certs\") pod \"kube-auth-proxy-5c4bb65c68-jwxlp\" (UID: \"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.147213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.147147 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqblf\" (UniqueName: \"kubernetes.io/projected/d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a-kube-api-access-wqblf\") pod \"kube-auth-proxy-5c4bb65c68-jwxlp\" (UID: \"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.204671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.204644 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" Oct 30 06:01:13.510531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.510454 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp"] Oct 30 06:01:13.513707 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:01:13.513679 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a62ddf_f59d_4c0e_bbc5_2c90a8517c6a.slice/crio-f8f73edc05b4d825ae386bcdb1bb916d1e22ba40b887cf558046cf41c9d220c0 WatchSource:0}: Error finding container f8f73edc05b4d825ae386bcdb1bb916d1e22ba40b887cf558046cf41c9d220c0: Status 404 returned error can't find the container with id f8f73edc05b4d825ae386bcdb1bb916d1e22ba40b887cf558046cf41c9d220c0 Oct 30 06:01:13.995696 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:13.995662 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" event={"ID":"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a","Type":"ContainerStarted","Data":"f8f73edc05b4d825ae386bcdb1bb916d1e22ba40b887cf558046cf41c9d220c0"} Oct 30 06:01:16.005348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:16.005315 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" event={"ID":"d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a","Type":"ContainerStarted","Data":"74b783b3f87c107d8284c68198b988245082a1d2d9528cdc11da95302eaf3dec"} Oct 30 06:01:16.065041 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:16.064999 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-jwxlp" podStartSLOduration=1.672040284 podStartE2EDuration="4.064984985s" totalImagesPullingTime="2.392944701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:16.064270189 +0000 UTC m=+484.870791934" watchObservedRunningTime="2026-10-30 06:01:16.064984985 +0000 UTC m=+484.871506732" Oct 30 06:01:22.350723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.350694 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b"] Oct 30 06:01:22.356291 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.356275 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.382082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.382062 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-dsdcz\"" Oct 30 06:01:22.385556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.385538 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 06:01:22.385624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.385582 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 06:01:22.410122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.410094 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 06:01:22.410994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.410977 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b"] Oct 30 06:01:22.477607 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.477582 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a879b8c8-3bc4-4f44-9691-e924c6b703a8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.477725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.477613 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtv54\" (UniqueName: \"kubernetes.io/projected/a879b8c8-3bc4-4f44-9691-e924c6b703a8-kube-api-access-wtv54\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.477725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.477644 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a879b8c8-3bc4-4f44-9691-e924c6b703a8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.477843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.477799 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a879b8c8-3bc4-4f44-9691-e924c6b703a8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.578089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.578061 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a879b8c8-3bc4-4f44-9691-e924c6b703a8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.578215 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.578098 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a879b8c8-3bc4-4f44-9691-e924c6b703a8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.578215 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.578136 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a879b8c8-3bc4-4f44-9691-e924c6b703a8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.578215 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.578155 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtv54\" (UniqueName: \"kubernetes.io/projected/a879b8c8-3bc4-4f44-9691-e924c6b703a8-kube-api-access-wtv54\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.578532 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.578510 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a879b8c8-3bc4-4f44-9691-e924c6b703a8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.578532 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.578525 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a879b8c8-3bc4-4f44-9691-e924c6b703a8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.579018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.578989 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a879b8c8-3bc4-4f44-9691-e924c6b703a8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.610071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.610031 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtv54\" (UniqueName: \"kubernetes.io/projected/a879b8c8-3bc4-4f44-9691-e924c6b703a8-kube-api-access-wtv54\") pod \"kserve-module-controller-manager-7566cf5fc8-mwq2b\" (UID: \"a879b8c8-3bc4-4f44-9691-e924c6b703a8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.667004 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.666980 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:22.981565 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:22.981532 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b"] Oct 30 06:01:22.984925 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:01:22.984896 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda879b8c8_3bc4_4f44_9691_e924c6b703a8.slice/crio-1641a574822969cb26f32aea60986545401fb31d09057df0fb5fe0cb46e8d1a4 WatchSource:0}: Error finding container 1641a574822969cb26f32aea60986545401fb31d09057df0fb5fe0cb46e8d1a4: Status 404 returned error can't find the container with id 1641a574822969cb26f32aea60986545401fb31d09057df0fb5fe0cb46e8d1a4 Oct 30 06:01:23.032024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:23.031993 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" event={"ID":"a879b8c8-3bc4-4f44-9691-e924c6b703a8","Type":"ContainerStarted","Data":"1641a574822969cb26f32aea60986545401fb31d09057df0fb5fe0cb46e8d1a4"} Oct 30 06:01:25.041831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:25.041795 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" event={"ID":"a879b8c8-3bc4-4f44-9691-e924c6b703a8","Type":"ContainerStarted","Data":"12a2537dea0ef2a48601d9118b889190eb8e09155ab2533d562acd2d14d237c8"} Oct 30 06:01:25.042238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:25.041923 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:01:25.118660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:25.118621 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" podStartSLOduration=1.63584978 podStartE2EDuration="3.118610184s" totalImagesPullingTime="1.482760404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:25.117723268 +0000 UTC m=+493.924245026" watchObservedRunningTime="2026-10-30 06:01:25.118610184 +0000 UTC m=+493.925131973" Oct 30 06:01:29.977063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:29.977030 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh"] Oct 30 06:01:29.980422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:29.980404 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.009950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.009925 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:01:30.009950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.009930 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 06:01:30.010107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.009935 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 06:01:30.015638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.015617 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 06:01:30.015638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.015618 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-72hgn\"" Oct 30 06:01:30.015777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.015671 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 06:01:30.026490 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.026468 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh"] Oct 30 06:01:30.032535 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.032512 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbvpv\" (UniqueName: \"kubernetes.io/projected/8614a2eb-b04f-43c6-acda-4f71032e93b6-kube-api-access-nbvpv\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.032638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.032557 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8614a2eb-b04f-43c6-acda-4f71032e93b6-cert\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.032638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.032583 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8614a2eb-b04f-43c6-acda-4f71032e93b6-metrics-certs\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.032722 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.032703 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8614a2eb-b04f-43c6-acda-4f71032e93b6-manager-config\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.133795 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.133769 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8614a2eb-b04f-43c6-acda-4f71032e93b6-cert\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.133927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.133802 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8614a2eb-b04f-43c6-acda-4f71032e93b6-metrics-certs\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.133927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.133825 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8614a2eb-b04f-43c6-acda-4f71032e93b6-manager-config\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.133927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.133886 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbvpv\" (UniqueName: \"kubernetes.io/projected/8614a2eb-b04f-43c6-acda-4f71032e93b6-kube-api-access-nbvpv\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.134592 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.134569 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8614a2eb-b04f-43c6-acda-4f71032e93b6-manager-config\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.136318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.136286 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8614a2eb-b04f-43c6-acda-4f71032e93b6-cert\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.136411 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.136325 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8614a2eb-b04f-43c6-acda-4f71032e93b6-metrics-certs\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.169284 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.169257 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbvpv\" (UniqueName: \"kubernetes.io/projected/8614a2eb-b04f-43c6-acda-4f71032e93b6-kube-api-access-nbvpv\") pod \"jobset-controller-manager-576fcc8bf6-kq8bh\" (UID: \"8614a2eb-b04f-43c6-acda-4f71032e93b6\") " pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.291798 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.291726 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:30.595869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:30.595840 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh"] Oct 30 06:01:30.598195 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:01:30.598158 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8614a2eb_b04f_43c6_acda_4f71032e93b6.slice/crio-b0fc2944acc54c2c1469893de1875ca1d97305970679838244c0cd1a43ed55a7 WatchSource:0}: Error finding container b0fc2944acc54c2c1469893de1875ca1d97305970679838244c0cd1a43ed55a7: Status 404 returned error can't find the container with id b0fc2944acc54c2c1469893de1875ca1d97305970679838244c0cd1a43ed55a7 Oct 30 06:01:31.063653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:31.063620 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" event={"ID":"8614a2eb-b04f-43c6-acda-4f71032e93b6","Type":"ContainerStarted","Data":"b0fc2944acc54c2c1469893de1875ca1d97305970679838244c0cd1a43ed55a7"} Oct 30 06:01:33.701343 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:01:33.701293 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:01:34.076975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:34.076897 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" event={"ID":"8614a2eb-b04f-43c6-acda-4f71032e93b6","Type":"ContainerStarted","Data":"e560cf7a47648f5dda1a78a121dfc3e063cd4658ea0392f6da4f2cef84bcf2b4"} Oct 30 06:01:34.077105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:34.077036 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:45.086023 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:45.085989 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" Oct 30 06:01:45.132998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:45.132947 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-576fcc8bf6-kq8bh" podStartSLOduration=13.42321078 podStartE2EDuration="16.132933606s" totalImagesPullingTime="2.709722826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:34.124671537 +0000 UTC m=+502.931193285" watchObservedRunningTime="2026-10-30 06:01:45.132933606 +0000 UTC m=+513.939455354" Oct 30 06:01:46.047388 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:01:46.047360 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwq2b" Oct 30 06:02:03.699457 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:02:03.699420 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:02:26.793770 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:26.793735 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr"] Oct 30 06:02:26.794217 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:26.793942 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" podUID="ea0d0110-19d2-428c-8234-dd9269d82d1e" containerName="discovery" containerID="cri-o://bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276" gracePeriod=30 Oct 30 06:02:26.805349 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:26.805322 4431 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-js6cr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.20:8080/ready\": dial tcp 10.133.0.20:8080: connect: connection refused" start-of-body= Oct 30 06:02:26.805477 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:26.805371 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" podUID="ea0d0110-19d2-428c-8234-dd9269d82d1e" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.20:8080/ready\": dial tcp 10.133.0.20:8080: connect: connection refused" Oct 30 06:02:27.034432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.034409 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:02:27.168000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.167927 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-kubeconfig\") pod \"ea0d0110-19d2-428c-8234-dd9269d82d1e\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " Oct 30 06:02:27.168000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.167970 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-kube-api-access-2nbjh\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-kube-api-access-2nbjh\") pod \"ea0d0110-19d2-428c-8234-dd9269d82d1e\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " Oct 30 06:02:27.168000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.168002 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-token\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-token\") pod \"ea0d0110-19d2-428c-8234-dd9269d82d1e\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " Oct 30 06:02:27.168276 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.168178 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-dns-cert\") pod \"ea0d0110-19d2-428c-8234-dd9269d82d1e\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " Oct 30 06:02:27.168276 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.168265 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0d0110-19d2-428c-8234-dd9269d82d1e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0d0110-19d2-428c-8234-dd9269d82d1e-local-certs\") pod \"ea0d0110-19d2-428c-8234-dd9269d82d1e\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " Oct 30 06:02:27.168429 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.168327 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-ca-configmap\") pod \"ea0d0110-19d2-428c-8234-dd9269d82d1e\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " Oct 30 06:02:27.168429 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.168380 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-cacerts\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-cacerts\") pod \"ea0d0110-19d2-428c-8234-dd9269d82d1e\" (UID: \"ea0d0110-19d2-428c-8234-dd9269d82d1e\") " Oct 30 06:02:27.168733 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.168681 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-ca-configmap" pod "ea0d0110-19d2-428c-8234-dd9269d82d1e" (UID: "ea0d0110-19d2-428c-8234-dd9269d82d1e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:27.170460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.170433 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-kubeconfig" pod "ea0d0110-19d2-428c-8234-dd9269d82d1e" (UID: "ea0d0110-19d2-428c-8234-dd9269d82d1e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:27.170571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.170508 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-cacerts" pod "ea0d0110-19d2-428c-8234-dd9269d82d1e" (UID: "ea0d0110-19d2-428c-8234-dd9269d82d1e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:27.170631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.170573 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-dns-cert" pod "ea0d0110-19d2-428c-8234-dd9269d82d1e" (UID: "ea0d0110-19d2-428c-8234-dd9269d82d1e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:27.170631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.170585 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-token" pod "ea0d0110-19d2-428c-8234-dd9269d82d1e" (UID: "ea0d0110-19d2-428c-8234-dd9269d82d1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:27.170631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.170585 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-kube-api-access-2nbjh" pod "ea0d0110-19d2-428c-8234-dd9269d82d1e" (UID: "ea0d0110-19d2-428c-8234-dd9269d82d1e"). InnerVolumeSpecName "kube-api-access-2nbjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:27.170938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.170921 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0d0110-19d2-428c-8234-dd9269d82d1e-local-certs" pod "ea0d0110-19d2-428c-8234-dd9269d82d1e" (UID: "ea0d0110-19d2-428c-8234-dd9269d82d1e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:27.269117 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.269093 4431 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-ca-configmap\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:27.269117 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.269114 4431 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-cacerts\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:27.269246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.269124 4431 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-kubeconfig\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:27.269246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.269133 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nbjh\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-kube-api-access-2nbjh\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:27.269246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.269141 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:27.269246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.269149 4431 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ea0d0110-19d2-428c-8234-dd9269d82d1e-istio-csr-dns-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:27.269246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.269157 4431 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0d0110-19d2-428c-8234-dd9269d82d1e-local-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:27.278799 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.278777 4431 generic.go:356] "Generic (PLEG): container finished" podID="ea0d0110-19d2-428c-8234-dd9269d82d1e" containerID="bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276" exitCode=0 Oct 30 06:02:27.278894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.278803 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" event={"ID":"ea0d0110-19d2-428c-8234-dd9269d82d1e","Type":"ContainerDied","Data":"bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276"} Oct 30 06:02:27.278894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.278847 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" Oct 30 06:02:27.278894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.278864 4431 scope.go:122] "RemoveContainer" containerID="bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276" Oct 30 06:02:27.279008 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.278853 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr" event={"ID":"ea0d0110-19d2-428c-8234-dd9269d82d1e","Type":"ContainerDied","Data":"cb0346a6d5924836a3d661dd68ba250cf1e791c07e5354cff2e9de8b48734172"} Oct 30 06:02:27.287972 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.287950 4431 scope.go:122] "RemoveContainer" containerID="bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276" Oct 30 06:02:27.288207 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:02:27.288186 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276\": container with ID starting with bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276 not found: ID does not exist" containerID="bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276" Oct 30 06:02:27.288250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.288215 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276"} err="failed to get container status \"bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276\": rpc error: code = NotFound desc = could not find container \"bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276\": container with ID starting with bceb4851f431041abc9c9219e1ae147d611adcbd0ce44845ea003893d7614276 not found: ID does not exist" Oct 30 06:02:27.362176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.362151 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr"] Oct 30 06:02:27.376841 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:27.376820 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-js6cr"] Oct 30 06:02:29.334585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:29.334551 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0d0110-19d2-428c-8234-dd9269d82d1e" path="/var/lib/kubelet/pods/ea0d0110-19d2-428c-8234-dd9269d82d1e/volumes" Oct 30 06:02:33.700292 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:02:33.700188 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:02:35.800451 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.800415 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp"] Oct 30 06:02:35.800900 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.800714 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ea0d0110-19d2-428c-8234-dd9269d82d1e" containerName="discovery" Oct 30 06:02:35.800900 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.800725 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0d0110-19d2-428c-8234-dd9269d82d1e" containerName="discovery" Oct 30 06:02:35.800900 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.800792 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0d0110-19d2-428c-8234-dd9269d82d1e" containerName="discovery" Oct 30 06:02:35.802659 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.802639 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:35.830621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.830591 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:02:35.830621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.830626 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:02:35.830774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.830624 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 06:02:35.830774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.830681 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 06:02:35.831728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.831704 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp"] Oct 30 06:02:35.931089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.931057 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-maas-api-tls\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:35.931089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.931091 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-openshift-service-ca\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:35.931263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.931114 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c9tk\" (UniqueName: \"kubernetes.io/projected/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-kube-api-access-9c9tk\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:35.931263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:35.931235 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-metrics-certs\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.032021 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.031989 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-maas-api-tls\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.032167 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.032023 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-openshift-service-ca\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.032167 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.032050 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c9tk\" (UniqueName: \"kubernetes.io/projected/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-kube-api-access-9c9tk\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.032167 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.032086 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-metrics-certs\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.032715 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.032694 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-openshift-service-ca\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.034520 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.034496 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-maas-api-tls\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.034603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.034504 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-metrics-certs\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.054643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.054599 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c9tk\" (UniqueName: \"kubernetes.io/projected/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-kube-api-access-9c9tk\") pod \"maas-api-868774c7cf-lwgjp\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.113991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.113967 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:36.414792 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:36.414763 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp"] Oct 30 06:02:36.416941 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:02:36.416917 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b48dcb_64ff_4f22_9be2_947564d1ab8e.slice/crio-bcfa69631f15b23553f8a13ba7181cfd2ec4b351b9020f757bb20d43e3d768aa WatchSource:0}: Error finding container bcfa69631f15b23553f8a13ba7181cfd2ec4b351b9020f757bb20d43e3d768aa: Status 404 returned error can't find the container with id bcfa69631f15b23553f8a13ba7181cfd2ec4b351b9020f757bb20d43e3d768aa Oct 30 06:02:37.321889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:37.321849 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" event={"ID":"b6b48dcb-64ff-4f22-9be2-947564d1ab8e","Type":"ContainerStarted","Data":"bcfa69631f15b23553f8a13ba7181cfd2ec4b351b9020f757bb20d43e3d768aa"} Oct 30 06:02:39.334369 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:39.334334 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" event={"ID":"b6b48dcb-64ff-4f22-9be2-947564d1ab8e","Type":"ContainerStarted","Data":"35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5"} Oct 30 06:02:39.334736 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:39.334384 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:39.377850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:39.377805 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" podStartSLOduration=2.127779447 podStartE2EDuration="4.377792178s" totalImagesPullingTime="2.250012731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:39.377027949 +0000 UTC m=+568.183549696" watchObservedRunningTime="2026-10-30 06:02:39.377792178 +0000 UTC m=+568.184313926" Oct 30 06:02:45.340633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:45.340599 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:46.332107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.332070 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj"] Oct 30 06:02:46.334142 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.334127 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.355515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.355492 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b4k9b\"" Oct 30 06:02:46.381600 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.381575 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj"] Oct 30 06:02:46.408723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.408693 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-metrics-certs\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.408865 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.408725 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-openshift-service-ca\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.408865 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.408764 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzv4\" (UniqueName: \"kubernetes.io/projected/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-kube-api-access-wdzv4\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.408979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.408888 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-maas-api-tls\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.509656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.509621 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzv4\" (UniqueName: \"kubernetes.io/projected/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-kube-api-access-wdzv4\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.509809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.509677 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-maas-api-tls\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.509809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.509721 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-metrics-certs\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.509809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.509739 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-openshift-service-ca\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.510384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.510342 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-openshift-service-ca\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.512283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.512259 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-metrics-certs\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.512380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.512277 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-maas-api-tls\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.537593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.537565 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzv4\" (UniqueName: \"kubernetes.io/projected/1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744-kube-api-access-wdzv4\") pod \"maas-api-897bbbb96-r6dwj\" (UID: \"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744\") " pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.644827 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.644765 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:46.950105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:46.950042 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj"] Oct 30 06:02:46.952807 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:02:46.952778 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa01b8a_fe13_40b6_8dfc_3d9ba9cf4744.slice/crio-b36e5981e2bb538824b74eb3c9b0a5923c43e114d741c2d8a5c6449afe629fc1 WatchSource:0}: Error finding container b36e5981e2bb538824b74eb3c9b0a5923c43e114d741c2d8a5c6449afe629fc1: Status 404 returned error can't find the container with id b36e5981e2bb538824b74eb3c9b0a5923c43e114d741c2d8a5c6449afe629fc1 Oct 30 06:02:47.361888 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:47.361852 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" event={"ID":"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744","Type":"ContainerStarted","Data":"c4f12f1addf4ad5528d33b2428f3bcfdadd548cdbc00b9af37d62b9bbf8b307a"} Oct 30 06:02:47.362274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:47.361909 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" event={"ID":"1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744","Type":"ContainerStarted","Data":"b36e5981e2bb538824b74eb3c9b0a5923c43e114d741c2d8a5c6449afe629fc1"} Oct 30 06:02:47.362274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:47.361956 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:47.408722 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:47.408676 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" podStartSLOduration=1.128533751 podStartE2EDuration="1.408660717s" totalImagesPullingTime="280.126966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:47.408283159 +0000 UTC m=+576.214804930" watchObservedRunningTime="2026-10-30 06:02:47.408660717 +0000 UTC m=+576.215182465" Oct 30 06:02:53.370902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.370871 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-897bbbb96-r6dwj" Oct 30 06:02:53.448977 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.448947 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp"] Oct 30 06:02:53.449193 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.449161 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" podUID="b6b48dcb-64ff-4f22-9be2-947564d1ab8e" containerName="maas-api" containerID="cri-o://35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5" gracePeriod=30 Oct 30 06:02:53.682690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.682666 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:53.765497 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.765467 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-openshift-service-ca\") pod \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " Oct 30 06:02:53.765623 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.765508 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-metrics-certs\") pod \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " Oct 30 06:02:53.765623 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.765557 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-kube-api-access-9c9tk\" (UniqueName: \"kubernetes.io/projected/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-kube-api-access-9c9tk\") pod \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " Oct 30 06:02:53.765623 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.765605 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-maas-api-tls\") pod \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\" (UID: \"b6b48dcb-64ff-4f22-9be2-947564d1ab8e\") " Oct 30 06:02:53.765826 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.765805 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-openshift-service-ca" pod "b6b48dcb-64ff-4f22-9be2-947564d1ab8e" (UID: "b6b48dcb-64ff-4f22-9be2-947564d1ab8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:53.767567 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.767543 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-metrics-certs" pod "b6b48dcb-64ff-4f22-9be2-947564d1ab8e" (UID: "b6b48dcb-64ff-4f22-9be2-947564d1ab8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:53.767690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.767568 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-maas-api-tls" pod "b6b48dcb-64ff-4f22-9be2-947564d1ab8e" (UID: "b6b48dcb-64ff-4f22-9be2-947564d1ab8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:53.767690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.767649 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-kube-api-access-9c9tk" pod "b6b48dcb-64ff-4f22-9be2-947564d1ab8e" (UID: "b6b48dcb-64ff-4f22-9be2-947564d1ab8e"). InnerVolumeSpecName "kube-api-access-9c9tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:53.866434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.866408 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:53.866434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.866434 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:53.866580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.866449 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c9tk\" (UniqueName: \"kubernetes.io/projected/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-kube-api-access-9c9tk\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:53.866580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:53.866462 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6b48dcb-64ff-4f22-9be2-947564d1ab8e-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:02:54.389084 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:54.389050 4431 generic.go:356] "Generic (PLEG): container finished" podID="b6b48dcb-64ff-4f22-9be2-947564d1ab8e" containerID="35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5" exitCode=0 Oct 30 06:02:54.389506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:54.389090 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" event={"ID":"b6b48dcb-64ff-4f22-9be2-947564d1ab8e","Type":"ContainerDied","Data":"35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5"} Oct 30 06:02:54.389506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:54.389128 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" event={"ID":"b6b48dcb-64ff-4f22-9be2-947564d1ab8e","Type":"ContainerDied","Data":"bcfa69631f15b23553f8a13ba7181cfd2ec4b351b9020f757bb20d43e3d768aa"} Oct 30 06:02:54.389506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:54.389149 4431 scope.go:122] "RemoveContainer" containerID="35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5" Oct 30 06:02:54.389506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:54.389094 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp" Oct 30 06:02:54.398098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:54.398084 4431 scope.go:122] "RemoveContainer" containerID="35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5" Oct 30 06:02:54.398389 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:02:54.398360 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5\": container with ID starting with 35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5 not found: ID does not exist" containerID="35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5" Oct 30 06:02:54.398487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:54.398400 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5"} err="failed to get container status \"35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5\": rpc error: code = NotFound desc = could not find container \"35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5\": container with ID starting with 35eb6487036cf69a14bf754ff63627369ecac263347da0943c361f0a1c66e7d5 not found: ID does not exist" Oct 30 06:02:54.438501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:54.438477 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp"] Oct 30 06:02:54.450550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:54.450526 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-lwgjp"] Oct 30 06:02:55.334844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:02:55.334812 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b48dcb-64ff-4f22-9be2-947564d1ab8e" path="/var/lib/kubelet/pods/b6b48dcb-64ff-4f22-9be2-947564d1ab8e/volumes" Oct 30 06:03:03.699675 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:03:03.699636 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:03:33.699720 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:03:33.699682 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:04:03.699723 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:04:03.699687 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:04:04.258746 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.258707 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-9z8hf"] Oct 30 06:04:04.259073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.259056 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b6b48dcb-64ff-4f22-9be2-947564d1ab8e" containerName="maas-api" Oct 30 06:04:04.259151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.259076 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b48dcb-64ff-4f22-9be2-947564d1ab8e" containerName="maas-api" Oct 30 06:04:04.259207 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.259162 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b48dcb-64ff-4f22-9be2-947564d1ab8e" containerName="maas-api" Oct 30 06:04:04.262000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.261980 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:04.283073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.283046 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-9z8hf"] Oct 30 06:04:04.287138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.287115 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxmx\" (UniqueName: \"kubernetes.io/projected/5bd14b2f-e551-4954-b174-bd801635f6e8-kube-api-access-ttxmx\") pod \"ai-gateway-controller-7d85675f98-9z8hf\" (UID: \"5bd14b2f-e551-4954-b174-bd801635f6e8\") " pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:04.287240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.287192 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-9z8hf\" (UID: \"5bd14b2f-e551-4954-b174-bd801635f6e8\") " pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:04.290782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.290761 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 06:04:04.290886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.290787 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 06:04:04.290886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.290863 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xdnrg\"" Oct 30 06:04:04.388640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.388606 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-9z8hf\" (UID: \"5bd14b2f-e551-4954-b174-bd801635f6e8\") " pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:04.388792 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.388677 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxmx\" (UniqueName: \"kubernetes.io/projected/5bd14b2f-e551-4954-b174-bd801635f6e8-kube-api-access-ttxmx\") pod \"ai-gateway-controller-7d85675f98-9z8hf\" (UID: \"5bd14b2f-e551-4954-b174-bd801635f6e8\") " pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:04.388792 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:04:04.388757 4431 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found Oct 30 06:04:04.388903 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:04:04.388817 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs podName:5bd14b2f-e551-4954-b174-bd801635f6e8 nodeName:}" failed. No retries permitted until 2026-10-30 06:04:04.888802342 +0000 UTC m=+653.695324073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs") pod "ai-gateway-controller-7d85675f98-9z8hf" (UID: "5bd14b2f-e551-4954-b174-bd801635f6e8") : secret "ai-gateway-controller-metrics-tls" not found Oct 30 06:04:04.411917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.411894 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxmx\" (UniqueName: \"kubernetes.io/projected/5bd14b2f-e551-4954-b174-bd801635f6e8-kube-api-access-ttxmx\") pod \"ai-gateway-controller-7d85675f98-9z8hf\" (UID: \"5bd14b2f-e551-4954-b174-bd801635f6e8\") " pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:04.891784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.891757 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-9z8hf\" (UID: \"5bd14b2f-e551-4954-b174-bd801635f6e8\") " pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:04.893944 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:04.893922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-9z8hf\" (UID: \"5bd14b2f-e551-4954-b174-bd801635f6e8\") " pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:05.174274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:05.174201 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:05.470587 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:05.470560 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-9z8hf"] Oct 30 06:04:05.473483 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:04:05.473443 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd14b2f_e551_4954_b174_bd801635f6e8.slice/crio-8bbeb46efc0d89b93fdb2e15794d9a2c7e44d711f323c8fb110f470e9e14b10f WatchSource:0}: Error finding container 8bbeb46efc0d89b93fdb2e15794d9a2c7e44d711f323c8fb110f470e9e14b10f: Status 404 returned error can't find the container with id 8bbeb46efc0d89b93fdb2e15794d9a2c7e44d711f323c8fb110f470e9e14b10f Oct 30 06:04:05.656250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:05.656211 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" event={"ID":"5bd14b2f-e551-4954-b174-bd801635f6e8","Type":"ContainerStarted","Data":"8bbeb46efc0d89b93fdb2e15794d9a2c7e44d711f323c8fb110f470e9e14b10f"} Oct 30 06:04:07.667616 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:07.667576 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" event={"ID":"5bd14b2f-e551-4954-b174-bd801635f6e8","Type":"ContainerStarted","Data":"b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966"} Oct 30 06:04:07.667994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:07.667709 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:07.711348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:07.711279 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" podStartSLOduration=2.12651351 podStartE2EDuration="3.711268843s" totalImagesPullingTime="1.584755333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:07.710942479 +0000 UTC m=+656.517464226" watchObservedRunningTime="2026-10-30 06:04:07.711268843 +0000 UTC m=+656.517790590" Oct 30 06:04:09.566088 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.566056 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-f26kj"] Oct 30 06:04:09.569395 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.569379 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:09.591867 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.591844 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8c4cj\"" Oct 30 06:04:09.604049 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.604022 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 06:04:09.605129 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.605108 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-f26kj"] Oct 30 06:04:09.628709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.628679 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9s2\" (UniqueName: \"kubernetes.io/projected/86cdedee-43d9-421a-9ffd-cfc6775eed66-kube-api-access-9h9s2\") pod \"payload-pre-processing-7b98bb678c-f26kj\" (UID: \"86cdedee-43d9-421a-9ffd-cfc6775eed66\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:09.628802 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.628716 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86cdedee-43d9-421a-9ffd-cfc6775eed66-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-f26kj\" (UID: \"86cdedee-43d9-421a-9ffd-cfc6775eed66\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:09.662694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.662666 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hpphs"] Oct 30 06:04:09.665756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.665742 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:09.690401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.690378 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hpphs"] Oct 30 06:04:09.729181 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.729156 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2cce36-89ca-4ea5-a488-be5af53dbf14-plugins-config-volume\") pod \"payload-processing-5f78687d5b-hpphs\" (UID: \"9b2cce36-89ca-4ea5-a488-be5af53dbf14\") " pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:09.729331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.729194 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86cdedee-43d9-421a-9ffd-cfc6775eed66-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-f26kj\" (UID: \"86cdedee-43d9-421a-9ffd-cfc6775eed66\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:09.729331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.729216 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tn5f\" (UniqueName: \"kubernetes.io/projected/9b2cce36-89ca-4ea5-a488-be5af53dbf14-kube-api-access-9tn5f\") pod \"payload-processing-5f78687d5b-hpphs\" (UID: \"9b2cce36-89ca-4ea5-a488-be5af53dbf14\") " pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:09.729435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.729421 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9s2\" (UniqueName: \"kubernetes.io/projected/86cdedee-43d9-421a-9ffd-cfc6775eed66-kube-api-access-9h9s2\") pod \"payload-pre-processing-7b98bb678c-f26kj\" (UID: \"86cdedee-43d9-421a-9ffd-cfc6775eed66\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:09.729886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.729867 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86cdedee-43d9-421a-9ffd-cfc6775eed66-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-f26kj\" (UID: \"86cdedee-43d9-421a-9ffd-cfc6775eed66\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:09.754117 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.754095 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9s2\" (UniqueName: \"kubernetes.io/projected/86cdedee-43d9-421a-9ffd-cfc6775eed66-kube-api-access-9h9s2\") pod \"payload-pre-processing-7b98bb678c-f26kj\" (UID: \"86cdedee-43d9-421a-9ffd-cfc6775eed66\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:09.830317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.830226 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2cce36-89ca-4ea5-a488-be5af53dbf14-plugins-config-volume\") pod \"payload-processing-5f78687d5b-hpphs\" (UID: \"9b2cce36-89ca-4ea5-a488-be5af53dbf14\") " pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:09.830317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.830270 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tn5f\" (UniqueName: \"kubernetes.io/projected/9b2cce36-89ca-4ea5-a488-be5af53dbf14-kube-api-access-9tn5f\") pod \"payload-processing-5f78687d5b-hpphs\" (UID: \"9b2cce36-89ca-4ea5-a488-be5af53dbf14\") " pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:09.830939 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.830922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2cce36-89ca-4ea5-a488-be5af53dbf14-plugins-config-volume\") pod \"payload-processing-5f78687d5b-hpphs\" (UID: \"9b2cce36-89ca-4ea5-a488-be5af53dbf14\") " pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:09.853807 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.853781 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tn5f\" (UniqueName: \"kubernetes.io/projected/9b2cce36-89ca-4ea5-a488-be5af53dbf14-kube-api-access-9tn5f\") pod \"payload-processing-5f78687d5b-hpphs\" (UID: \"9b2cce36-89ca-4ea5-a488-be5af53dbf14\") " pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:09.881135 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.881107 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:09.979042 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:09.979014 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:10.177745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:10.177720 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-f26kj"] Oct 30 06:04:10.180099 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:04:10.180070 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cdedee_43d9_421a_9ffd_cfc6775eed66.slice/crio-89037c1e5bcbb993aa23f742ebce3a9c0a47f3e996b781ab2bcae76b899a9e35 WatchSource:0}: Error finding container 89037c1e5bcbb993aa23f742ebce3a9c0a47f3e996b781ab2bcae76b899a9e35: Status 404 returned error can't find the container with id 89037c1e5bcbb993aa23f742ebce3a9c0a47f3e996b781ab2bcae76b899a9e35 Oct 30 06:04:10.276680 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:10.276642 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hpphs"] Oct 30 06:04:10.278793 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:04:10.278766 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2cce36_89ca_4ea5_a488_be5af53dbf14.slice/crio-413b1e9871b045e1d909a6caca1a072a423c9741c9e50014b3f3bfa3b17fe93a WatchSource:0}: Error finding container 413b1e9871b045e1d909a6caca1a072a423c9741c9e50014b3f3bfa3b17fe93a: Status 404 returned error can't find the container with id 413b1e9871b045e1d909a6caca1a072a423c9741c9e50014b3f3bfa3b17fe93a Oct 30 06:04:10.682694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:10.682658 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" event={"ID":"9b2cce36-89ca-4ea5-a488-be5af53dbf14","Type":"ContainerStarted","Data":"413b1e9871b045e1d909a6caca1a072a423c9741c9e50014b3f3bfa3b17fe93a"} Oct 30 06:04:10.684094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:10.684069 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" event={"ID":"86cdedee-43d9-421a-9ffd-cfc6775eed66","Type":"ContainerStarted","Data":"89037c1e5bcbb993aa23f742ebce3a9c0a47f3e996b781ab2bcae76b899a9e35"} Oct 30 06:04:12.694471 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:12.694427 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" event={"ID":"9b2cce36-89ca-4ea5-a488-be5af53dbf14","Type":"ContainerStarted","Data":"446de045f0f67dfcd0f96cb80012d8cdf6472eb941ef83d8f8fa48d503cf6c45"} Oct 30 06:04:12.694899 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:12.694496 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:12.695856 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:12.695832 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" event={"ID":"86cdedee-43d9-421a-9ffd-cfc6775eed66","Type":"ContainerStarted","Data":"af284ebdb99565e1d4e7637a39c9189344489c8928d461a8ec839f7b3ce80cde"} Oct 30 06:04:12.695965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:12.695952 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:12.736724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:12.736681 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" podStartSLOduration=1.590639943 podStartE2EDuration="3.736669267s" totalImagesPullingTime="2.146029324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:12.73550422 +0000 UTC m=+661.542025969" watchObservedRunningTime="2026-10-30 06:04:12.736669267 +0000 UTC m=+661.543191071" Oct 30 06:04:12.781383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:12.781342 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" podStartSLOduration=1.5817665619999999 podStartE2EDuration="3.781327381s" totalImagesPullingTime="2.199560819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:12.7804274 +0000 UTC m=+661.586949148" watchObservedRunningTime="2026-10-30 06:04:12.781327381 +0000 UTC m=+661.587849130" Oct 30 06:04:18.674371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:18.674336 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:18.958974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:18.958945 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6577b95894-swx2g"] Oct 30 06:04:18.962430 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:18.962414 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:18.982687 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:18.982663 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6577b95894-swx2g"] Oct 30 06:04:19.007465 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:19.007433 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bfe7f12-7c1e-4e56-8f6d-b94527d0428e-metrics-certs\") pod \"ai-gateway-controller-6577b95894-swx2g\" (UID: \"7bfe7f12-7c1e-4e56-8f6d-b94527d0428e\") " pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:19.007576 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:19.007495 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45q5\" (UniqueName: \"kubernetes.io/projected/7bfe7f12-7c1e-4e56-8f6d-b94527d0428e-kube-api-access-t45q5\") pod \"ai-gateway-controller-6577b95894-swx2g\" (UID: \"7bfe7f12-7c1e-4e56-8f6d-b94527d0428e\") " pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:19.108171 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:19.108148 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bfe7f12-7c1e-4e56-8f6d-b94527d0428e-metrics-certs\") pod \"ai-gateway-controller-6577b95894-swx2g\" (UID: \"7bfe7f12-7c1e-4e56-8f6d-b94527d0428e\") " pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:19.108265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:19.108187 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45q5\" (UniqueName: \"kubernetes.io/projected/7bfe7f12-7c1e-4e56-8f6d-b94527d0428e-kube-api-access-t45q5\") pod \"ai-gateway-controller-6577b95894-swx2g\" (UID: \"7bfe7f12-7c1e-4e56-8f6d-b94527d0428e\") " pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:19.110651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:19.110627 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bfe7f12-7c1e-4e56-8f6d-b94527d0428e-metrics-certs\") pod \"ai-gateway-controller-6577b95894-swx2g\" (UID: \"7bfe7f12-7c1e-4e56-8f6d-b94527d0428e\") " pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:19.131217 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:19.131193 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45q5\" (UniqueName: \"kubernetes.io/projected/7bfe7f12-7c1e-4e56-8f6d-b94527d0428e-kube-api-access-t45q5\") pod \"ai-gateway-controller-6577b95894-swx2g\" (UID: \"7bfe7f12-7c1e-4e56-8f6d-b94527d0428e\") " pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:19.274887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:19.274815 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:19.570099 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:19.569943 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6577b95894-swx2g"] Oct 30 06:04:19.572741 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:04:19.572713 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfe7f12_7c1e_4e56_8f6d_b94527d0428e.slice/crio-16154dee07f522afb7294f78843fb06e31adafe5f5319ffb31edcaf38d0841f2 WatchSource:0}: Error finding container 16154dee07f522afb7294f78843fb06e31adafe5f5319ffb31edcaf38d0841f2: Status 404 returned error can't find the container with id 16154dee07f522afb7294f78843fb06e31adafe5f5319ffb31edcaf38d0841f2 Oct 30 06:04:19.723107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:19.723075 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" event={"ID":"7bfe7f12-7c1e-4e56-8f6d-b94527d0428e","Type":"ContainerStarted","Data":"16154dee07f522afb7294f78843fb06e31adafe5f5319ffb31edcaf38d0841f2"} Oct 30 06:04:20.729230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:20.729191 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" event={"ID":"7bfe7f12-7c1e-4e56-8f6d-b94527d0428e","Type":"ContainerStarted","Data":"2e91f6abde75c8321432b79a50cc5bef4e2523b32a400346baba9e841d9c0133"} Oct 30 06:04:20.729611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:20.729334 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:20.770255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:20.770204 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" podStartSLOduration=2.458249325 podStartE2EDuration="2.770190824s" totalImagesPullingTime="311.941499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:20.768910133 +0000 UTC m=+669.575431904" watchObservedRunningTime="2026-10-30 06:04:20.770190824 +0000 UTC m=+669.576712572" Oct 30 06:04:23.702438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:23.702363 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f26kj" Oct 30 06:04:23.702438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:23.702423 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-hpphs" Oct 30 06:04:31.735888 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:31.735856 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6577b95894-swx2g" Oct 30 06:04:31.806548 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:31.806511 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-9z8hf"] Oct 30 06:04:31.806787 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:31.806753 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" podUID="5bd14b2f-e551-4954-b174-bd801635f6e8" containerName="manager" containerID="cri-o://b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966" gracePeriod=30 Oct 30 06:04:32.040216 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.040191 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:32.101904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.101878 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs\") pod \"5bd14b2f-e551-4954-b174-bd801635f6e8\" (UID: \"5bd14b2f-e551-4954-b174-bd801635f6e8\") " Oct 30 06:04:32.102013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.101926 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd14b2f-e551-4954-b174-bd801635f6e8-kube-api-access-ttxmx\" (UniqueName: \"kubernetes.io/projected/5bd14b2f-e551-4954-b174-bd801635f6e8-kube-api-access-ttxmx\") pod \"5bd14b2f-e551-4954-b174-bd801635f6e8\" (UID: \"5bd14b2f-e551-4954-b174-bd801635f6e8\") " Oct 30 06:04:32.104100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.104072 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs" pod "5bd14b2f-e551-4954-b174-bd801635f6e8" (UID: "5bd14b2f-e551-4954-b174-bd801635f6e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:04:32.104196 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.104069 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd14b2f-e551-4954-b174-bd801635f6e8-kube-api-access-ttxmx" pod "5bd14b2f-e551-4954-b174-bd801635f6e8" (UID: "5bd14b2f-e551-4954-b174-bd801635f6e8"). InnerVolumeSpecName "kube-api-access-ttxmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:32.202856 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.202821 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bd14b2f-e551-4954-b174-bd801635f6e8-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:04:32.202856 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.202851 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttxmx\" (UniqueName: \"kubernetes.io/projected/5bd14b2f-e551-4954-b174-bd801635f6e8-kube-api-access-ttxmx\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:04:32.779017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.778977 4431 generic.go:356] "Generic (PLEG): container finished" podID="5bd14b2f-e551-4954-b174-bd801635f6e8" containerID="b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966" exitCode=0 Oct 30 06:04:32.779541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.779030 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" Oct 30 06:04:32.779541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.779063 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" event={"ID":"5bd14b2f-e551-4954-b174-bd801635f6e8","Type":"ContainerDied","Data":"b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966"} Oct 30 06:04:32.779541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.779134 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-9z8hf" event={"ID":"5bd14b2f-e551-4954-b174-bd801635f6e8","Type":"ContainerDied","Data":"8bbeb46efc0d89b93fdb2e15794d9a2c7e44d711f323c8fb110f470e9e14b10f"} Oct 30 06:04:32.779541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.779166 4431 scope.go:122] "RemoveContainer" containerID="b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966" Oct 30 06:04:32.787442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.787425 4431 scope.go:122] "RemoveContainer" containerID="b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966" Oct 30 06:04:32.787704 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:04:32.787687 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966\": container with ID starting with b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966 not found: ID does not exist" containerID="b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966" Oct 30 06:04:32.787755 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.787714 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966"} err="failed to get container status \"b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966\": rpc error: code = NotFound desc = could not find container \"b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966\": container with ID starting with b1aa3d7c84f80b294061e929542d00204c334e5afaf23f128dff061ebba0a966 not found: ID does not exist" Oct 30 06:04:32.828504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.828477 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-9z8hf"] Oct 30 06:04:32.840318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:32.840276 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-9z8hf"] Oct 30 06:04:33.335722 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:33.335690 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd14b2f-e551-4954-b174-bd801635f6e8" path="/var/lib/kubelet/pods/5bd14b2f-e551-4954-b174-bd801635f6e8/volumes" Oct 30 06:04:33.699357 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:04:33.699326 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:04:51.006902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.006864 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd"] Oct 30 06:04:51.007402 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.007372 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5bd14b2f-e551-4954-b174-bd801635f6e8" containerName="manager" Oct 30 06:04:51.007402 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.007403 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd14b2f-e551-4954-b174-bd801635f6e8" containerName="manager" Oct 30 06:04:51.007506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.007491 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd14b2f-e551-4954-b174-bd801635f6e8" containerName="manager" Oct 30 06:04:51.011956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.011937 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.041728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.041710 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 06:04:51.041728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.041729 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8c689\"" Oct 30 06:04:51.043714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.043692 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd"] Oct 30 06:04:51.058486 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.058467 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 06:04:51.058597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.058545 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 06:04:51.140313 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.140283 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.140465 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.140337 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.140465 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.140412 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.140565 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.140478 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2p2c\" (UniqueName: \"kubernetes.io/projected/322f9a55-29c0-4cfb-8f2f-a736321f7b45-kube-api-access-c2p2c\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.140565 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.140500 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322f9a55-29c0-4cfb-8f2f-a736321f7b45-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.140652 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.140583 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.241709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.241680 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.241834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.241719 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.241834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.241746 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.241834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.241788 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2p2c\" (UniqueName: \"kubernetes.io/projected/322f9a55-29c0-4cfb-8f2f-a736321f7b45-kube-api-access-c2p2c\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.241834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.241819 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322f9a55-29c0-4cfb-8f2f-a736321f7b45-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.242043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.241852 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.242111 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.242061 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.242176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.242131 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.242234 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.242209 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.244002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.243979 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322f9a55-29c0-4cfb-8f2f-a736321f7b45-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.244246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.244228 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322f9a55-29c0-4cfb-8f2f-a736321f7b45-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.268886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.268831 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2p2c\" (UniqueName: \"kubernetes.io/projected/322f9a55-29c0-4cfb-8f2f-a736321f7b45-kube-api-access-c2p2c\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd\" (UID: \"322f9a55-29c0-4cfb-8f2f-a736321f7b45\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.324996 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.324965 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:51.629595 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.629537 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd"] Oct 30 06:04:51.631760 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:04:51.631735 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322f9a55_29c0_4cfb_8f2f_a736321f7b45.slice/crio-b83d8c95dee038671e70287d3bcb48da956da5382bfdeb7a5ca83774f5f02cf4 WatchSource:0}: Error finding container b83d8c95dee038671e70287d3bcb48da956da5382bfdeb7a5ca83774f5f02cf4: Status 404 returned error can't find the container with id b83d8c95dee038671e70287d3bcb48da956da5382bfdeb7a5ca83774f5f02cf4 Oct 30 06:04:51.857136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:51.857103 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" event={"ID":"322f9a55-29c0-4cfb-8f2f-a736321f7b45","Type":"ContainerStarted","Data":"b83d8c95dee038671e70287d3bcb48da956da5382bfdeb7a5ca83774f5f02cf4"} Oct 30 06:04:53.866287 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:53.866253 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" event={"ID":"322f9a55-29c0-4cfb-8f2f-a736321f7b45","Type":"ContainerStarted","Data":"2fde978b6d95ad229048f2a065674874cb5192979cd335d273c29c880814567c"} Oct 30 06:04:53.909736 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:53.909692 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" podStartSLOduration=2.716150645 podStartE2EDuration="3.909677825s" totalImagesPullingTime="1.19352718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:53.908014314 +0000 UTC m=+702.714536065" watchObservedRunningTime="2026-10-30 06:04:53.909677825 +0000 UTC m=+702.716199572" Oct 30 06:04:54.325538 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:54.325513 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:04:58.148203 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.148164 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk"] Oct 30 06:04:58.152138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.152115 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.178377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.178354 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 06:04:58.180149 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.180122 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk"] Oct 30 06:04:58.302072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.302038 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.302072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.302073 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.302270 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.302094 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.302270 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.302128 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227f9\" (UniqueName: \"kubernetes.io/projected/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-kube-api-access-227f9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.302270 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.302246 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.302398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.302281 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.403393 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.403297 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.403393 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.403376 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227f9\" (UniqueName: \"kubernetes.io/projected/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-kube-api-access-227f9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.403628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.403409 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.403628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.403426 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.403628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.403491 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.403628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.403517 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.403891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.403865 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.403951 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.403924 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.404043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.404022 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.405699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.405678 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.405831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.405812 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.426517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.426494 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227f9\" (UniqueName: \"kubernetes.io/projected/642ea77d-6dc0-4e6e-9a79-d5056b3d1589-kube-api-access-227f9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q92kk\" (UID: \"642ea77d-6dc0-4e6e-9a79-d5056b3d1589\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.465949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.465923 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:04:58.776126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.776092 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk"] Oct 30 06:04:58.779252 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:04:58.779221 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642ea77d_6dc0_4e6e_9a79_d5056b3d1589.slice/crio-70c59e50b4d7205b39c0cf850d2a00a7891597645e976528fd7c99b22228de03 WatchSource:0}: Error finding container 70c59e50b4d7205b39c0cf850d2a00a7891597645e976528fd7c99b22228de03: Status 404 returned error can't find the container with id 70c59e50b4d7205b39c0cf850d2a00a7891597645e976528fd7c99b22228de03 Oct 30 06:04:58.887362 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:58.887325 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" event={"ID":"642ea77d-6dc0-4e6e-9a79-d5056b3d1589","Type":"ContainerStarted","Data":"70c59e50b4d7205b39c0cf850d2a00a7891597645e976528fd7c99b22228de03"} Oct 30 06:04:59.892542 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:59.892499 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" event={"ID":"642ea77d-6dc0-4e6e-9a79-d5056b3d1589","Type":"ContainerStarted","Data":"082a0754a42ed3381e61090f73ca6b9ed21d666655cdf82298bcb3212e52353f"} Oct 30 06:04:59.935644 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:04:59.935591 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" podStartSLOduration=1.7213664039999999 podStartE2EDuration="1.93557758s" totalImagesPullingTime="214.211176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:59.935155476 +0000 UTC m=+708.741677225" watchObservedRunningTime="2026-10-30 06:04:59.93557758 +0000 UTC m=+708.742099329" Oct 30 06:05:00.466251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.466212 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:05:00.490343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.489178 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s"] Oct 30 06:05:00.492466 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.492446 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.513984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.513962 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 06:05:00.526545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.526520 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s"] Oct 30 06:05:00.622854 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.622818 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkzs\" (UniqueName: \"kubernetes.io/projected/fd9f37fd-8bf5-4c25-be74-c36945d782eb-kube-api-access-9dkzs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.623016 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.622871 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.623016 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.622954 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd9f37fd-8bf5-4c25-be74-c36945d782eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.623016 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.623005 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.623117 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.623088 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.623149 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.623123 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.723960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.723883 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.723960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.723920 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd9f37fd-8bf5-4c25-be74-c36945d782eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.723960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.723952 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.724267 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.723979 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.724267 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.724003 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.724267 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.724059 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkzs\" (UniqueName: \"kubernetes.io/projected/fd9f37fd-8bf5-4c25-be74-c36945d782eb-kube-api-access-9dkzs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.724461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.724425 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.724560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.724488 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.724613 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.724590 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.726211 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.726189 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd9f37fd-8bf5-4c25-be74-c36945d782eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.726551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.726535 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd9f37fd-8bf5-4c25-be74-c36945d782eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.747068 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.747041 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkzs\" (UniqueName: \"kubernetes.io/projected/fd9f37fd-8bf5-4c25-be74-c36945d782eb-kube-api-access-9dkzs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ds65s\" (UID: \"fd9f37fd-8bf5-4c25-be74-c36945d782eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:00.806321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:00.806272 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:01.103429 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:01.103260 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s"] Oct 30 06:05:01.105887 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:05:01.105860 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9f37fd_8bf5_4c25_be74_c36945d782eb.slice/crio-ce64eada9b98b7dcf4e8c79c2e79a5720303dcdbe7d2275514025610ed522f11 WatchSource:0}: Error finding container ce64eada9b98b7dcf4e8c79c2e79a5720303dcdbe7d2275514025610ed522f11: Status 404 returned error can't find the container with id ce64eada9b98b7dcf4e8c79c2e79a5720303dcdbe7d2275514025610ed522f11 Oct 30 06:05:01.325940 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:01.325875 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:05:01.340100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:01.340078 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:05:01.902082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:01.902044 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" event={"ID":"fd9f37fd-8bf5-4c25-be74-c36945d782eb","Type":"ContainerStarted","Data":"e9e5fa45e897e92c99d43033d618f4e84b93c363c6caba9726e092e8c5abb8d9"} Oct 30 06:05:01.902267 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:01.902140 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" event={"ID":"fd9f37fd-8bf5-4c25-be74-c36945d782eb","Type":"ContainerStarted","Data":"ce64eada9b98b7dcf4e8c79c2e79a5720303dcdbe7d2275514025610ed522f11"} Oct 30 06:05:01.913663 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:01.913636 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd" Oct 30 06:05:01.944823 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:01.944768 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" podStartSLOduration=1.724962704 podStartE2EDuration="1.944750672s" totalImagesPullingTime="219.787968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:01.942325579 +0000 UTC m=+710.748847335" watchObservedRunningTime="2026-10-30 06:05:01.944750672 +0000 UTC m=+710.751272422" Oct 30 06:05:02.425633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.425596 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w"] Oct 30 06:05:02.429487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.429465 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.444168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.444148 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 06:05:02.457067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.457046 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w"] Oct 30 06:05:02.538081 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.538048 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.538236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.538090 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.538236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.538110 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.538236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.538176 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692vg\" (UniqueName: \"kubernetes.io/projected/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-kube-api-access-692vg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.538368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.538237 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.538368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.538261 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.639517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.639485 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.639517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.639521 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.639759 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.639543 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692vg\" (UniqueName: \"kubernetes.io/projected/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-kube-api-access-692vg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.639759 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.639587 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.639759 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.639607 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.639759 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.639674 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.639984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.639938 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.639984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.639960 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.640086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.640034 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.641829 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.641806 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.642076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.642057 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.664641 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.664618 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-692vg\" (UniqueName: \"kubernetes.io/projected/292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8-kube-api-access-692vg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-l7s2w\" (UID: \"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.742739 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.742719 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:02.806653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:02.806629 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:03.039946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:03.039781 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w"] Oct 30 06:05:03.042652 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:05:03.042627 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292e4b1e_2d8a_42a0_b26c_b4de0c2e44f8.slice/crio-520e10f1ae723dc334eb5c70b0d23199497a47d297ae4602d656da91ebe95ea4 WatchSource:0}: Error finding container 520e10f1ae723dc334eb5c70b0d23199497a47d297ae4602d656da91ebe95ea4: Status 404 returned error can't find the container with id 520e10f1ae723dc334eb5c70b0d23199497a47d297ae4602d656da91ebe95ea4 Oct 30 06:05:03.699264 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:05:03.699228 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:05:03.911549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:03.911510 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" event={"ID":"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8","Type":"ContainerStarted","Data":"014bb4e9254056a603a123d989515dab062afec430f97b817c953a2bd3de16b9"} Oct 30 06:05:03.911713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:03.911562 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" event={"ID":"292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8","Type":"ContainerStarted","Data":"520e10f1ae723dc334eb5c70b0d23199497a47d297ae4602d656da91ebe95ea4"} Oct 30 06:05:03.955068 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:03.954981 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" podStartSLOduration=1.786927326 podStartE2EDuration="1.954968581s" totalImagesPullingTime="168.041255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:03.953323973 +0000 UTC m=+712.759845712" watchObservedRunningTime="2026-10-30 06:05:03.954968581 +0000 UTC m=+712.761490329" Oct 30 06:05:04.743343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:04.743284 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:05.456238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.456206 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh"] Oct 30 06:05:05.460027 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.460008 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.487461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.487439 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 06:05:05.488958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.488936 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh"] Oct 30 06:05:05.563945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.563920 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.564075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.563978 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.564113 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.564079 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxtr\" (UniqueName: \"kubernetes.io/projected/bf2fe315-4bfd-436b-849e-ce063c496de2-kube-api-access-qsxtr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.564184 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.564157 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.564219 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.564209 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2fe315-4bfd-436b-849e-ce063c496de2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.564249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.564236 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.665695 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.665670 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2fe315-4bfd-436b-849e-ce063c496de2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.665840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.665705 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.665840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.665769 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.665921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.665865 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.665921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.665902 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxtr\" (UniqueName: \"kubernetes.io/projected/bf2fe315-4bfd-436b-849e-ce063c496de2-kube-api-access-qsxtr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.666012 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.665924 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.666273 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.666234 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.666401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.666240 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.666401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.666349 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.667997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.667970 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2fe315-4bfd-436b-849e-ce063c496de2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.668152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.668135 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2fe315-4bfd-436b-849e-ce063c496de2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.688541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.688518 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxtr\" (UniqueName: \"kubernetes.io/projected/bf2fe315-4bfd-436b-849e-ce063c496de2-kube-api-access-qsxtr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh\" (UID: \"bf2fe315-4bfd-436b-849e-ce063c496de2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:05.773613 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:05.773549 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:06.073947 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:06.073922 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh"] Oct 30 06:05:06.081495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:06.081460 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:05:06.925838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:06.925798 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" event={"ID":"bf2fe315-4bfd-436b-849e-ce063c496de2","Type":"ContainerStarted","Data":"aae2d43142693abdc303150d640641f699754dd41b2e5e015661436161295171"} Oct 30 06:05:06.926272 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:06.925852 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" event={"ID":"bf2fe315-4bfd-436b-849e-ce063c496de2","Type":"ContainerStarted","Data":"a09dc375e55eb0ccff2d4a13505d78003e5d1de9b70b3e053f6ac95a80ef63f0"} Oct 30 06:05:06.968261 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:06.968209 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" podStartSLOduration=1.7777538050000001 podStartE2EDuration="1.968193276s" totalImagesPullingTime="190.439471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:06.967782364 +0000 UTC m=+715.774304113" watchObservedRunningTime="2026-10-30 06:05:06.968193276 +0000 UTC m=+715.774715024" Oct 30 06:05:07.773978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:07.773944 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:08.466413 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:08.466377 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:05:08.478492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:08.478469 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:05:08.945933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:08.945907 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q92kk" Oct 30 06:05:09.791820 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.791792 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr"] Oct 30 06:05:09.795984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.795958 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:09.813827 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.813803 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hc86w\"" Oct 30 06:05:09.825137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.825117 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr"] Oct 30 06:05:09.825926 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.825905 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 06:05:09.898955 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.898923 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:09.899109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.898964 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:09.899109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.898984 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:09.899194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.899129 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:09.899194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.899154 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98ee5c8-0953-4edf-8207-3db4f00c709b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:09.899194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.899183 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72n7r\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-kube-api-access-72n7r\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:09.899297 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.899242 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:09.899364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.899341 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:09.899430 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:09.899418 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000404 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72n7r\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-kube-api-access-72n7r\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000441 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000458 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000565 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000601 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000641 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000666 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000739 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000762 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98ee5c8-0953-4edf-8207-3db4f00c709b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.000868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.000858 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.001067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.001031 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.001184 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.001149 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.001330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.001239 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.001534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.001516 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98ee5c8-0953-4edf-8207-3db4f00c709b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.002957 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.002935 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.003033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.002995 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.023377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.023358 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72n7r\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-kube-api-access-72n7r\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.029424 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.029404 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5mhqr\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.108547 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.108496 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:10.428784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.428760 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr"] Oct 30 06:05:10.431201 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:05:10.431172 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98ee5c8_0953_4edf_8207_3db4f00c709b.slice/crio-fd16d11b980471dc00c26b153d320f13ac7bcb0c9bde464f619642a54cd01a35 WatchSource:0}: Error finding container fd16d11b980471dc00c26b153d320f13ac7bcb0c9bde464f619642a54cd01a35: Status 404 returned error can't find the container with id fd16d11b980471dc00c26b153d320f13ac7bcb0c9bde464f619642a54cd01a35 Oct 30 06:05:10.806668 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.806631 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:10.818940 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.818915 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:10.946055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.946017 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" event={"ID":"f98ee5c8-0953-4edf-8207-3db4f00c709b","Type":"ContainerStarted","Data":"fd16d11b980471dc00c26b153d320f13ac7bcb0c9bde464f619642a54cd01a35"} Oct 30 06:05:10.957183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:10.957157 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ds65s" Oct 30 06:05:11.409635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.409605 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp"] Oct 30 06:05:11.414536 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.414501 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.436538 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.436517 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 06:05:11.449705 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.449682 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp"] Oct 30 06:05:11.515243 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.515210 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.515433 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.515261 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnfk\" (UniqueName: \"kubernetes.io/projected/4199dace-66ce-4982-9ff6-629345b077ba-kube-api-access-7cnfk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.515433 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.515296 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.515537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.515423 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.515537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.515494 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.515627 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.515550 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4199dace-66ce-4982-9ff6-629345b077ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.616549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.616520 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4199dace-66ce-4982-9ff6-629345b077ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.616713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.616587 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.616713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.616634 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnfk\" (UniqueName: \"kubernetes.io/projected/4199dace-66ce-4982-9ff6-629345b077ba-kube-api-access-7cnfk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.616713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.616680 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.616713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.616704 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.616916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.616748 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.617095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.617060 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.617169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.617098 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.617169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.617132 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.619201 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.619178 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4199dace-66ce-4982-9ff6-629345b077ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.619404 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.619379 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4199dace-66ce-4982-9ff6-629345b077ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.644926 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.644864 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnfk\" (UniqueName: \"kubernetes.io/projected/4199dace-66ce-4982-9ff6-629345b077ba-kube-api-access-7cnfk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp\" (UID: \"4199dace-66ce-4982-9ff6-629345b077ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:11.737578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:11.737551 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:12.049611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:12.047979 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp"] Oct 30 06:05:12.051169 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:05:12.051140 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4199dace_66ce_4982_9ff6_629345b077ba.slice/crio-44e240006622008462e892732a77756a81450bd22cc4f6d950135b0586046114 WatchSource:0}: Error finding container 44e240006622008462e892732a77756a81450bd22cc4f6d950135b0586046114: Status 404 returned error can't find the container with id 44e240006622008462e892732a77756a81450bd22cc4f6d950135b0586046114 Oct 30 06:05:12.743367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:12.743320 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:12.757107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:12.757083 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:12.960485 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:12.960441 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" event={"ID":"4199dace-66ce-4982-9ff6-629345b077ba","Type":"ContainerStarted","Data":"0ac4d1ca02784e21b9845b2b9ad792affd12a0429e3cea845dadde1b2e6ae6bf"} Oct 30 06:05:12.960653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:12.960507 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" event={"ID":"4199dace-66ce-4982-9ff6-629345b077ba","Type":"ContainerStarted","Data":"44e240006622008462e892732a77756a81450bd22cc4f6d950135b0586046114"} Oct 30 06:05:12.972478 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:12.972453 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-l7s2w" Oct 30 06:05:13.002311 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:13.002068 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" podStartSLOduration=1.810972618 podStartE2EDuration="2.002047643s" totalImagesPullingTime="191.075025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:13.001473366 +0000 UTC m=+721.807995116" watchObservedRunningTime="2026-10-30 06:05:13.002047643 +0000 UTC m=+721.808569406" Oct 30 06:05:13.132973 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:13.132940 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:05:13.133386 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:13.133019 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:05:13.133386 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:13.133074 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:05:13.738663 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:13.738627 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:13.986545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:13.986501 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" event={"ID":"f98ee5c8-0953-4edf-8207-3db4f00c709b","Type":"ContainerStarted","Data":"edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c"} Oct 30 06:05:14.031749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:14.031653 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podStartSLOduration=2.332651695 podStartE2EDuration="5.031639235s" totalImagesPullingTime="2.69898754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:14.030522792 +0000 UTC m=+722.837044574" watchObservedRunningTime="2026-10-30 06:05:14.031639235 +0000 UTC m=+722.838160983" Oct 30 06:05:14.109456 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:14.109424 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:14.110991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:14.110965 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 06:05:14.111120 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:14.111020 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 06:05:15.109635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:15.109602 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 06:05:15.110117 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:15.109659 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 06:05:15.779128 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:15.778438 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:15.795516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:15.795488 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:16.006117 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:16.006087 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh" Oct 30 06:05:16.109168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:16.109093 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 06:05:16.109168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:16.109148 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 06:05:17.109134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:17.109094 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 06:05:17.109542 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:17.109159 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 06:05:18.109386 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:18.109353 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 06:05:18.109758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:18.109415 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 06:05:19.109742 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:19.109697 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 06:05:19.110111 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:19.109774 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 06:05:20.109863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:20.109821 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 06:05:20.110254 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:20.109896 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 06:05:21.738462 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:21.738430 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:21.750718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:21.750693 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:22.033953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:22.033882 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:05:22.150431 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:22.150393 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:05:22.150571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:22.150441 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:22.781042 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:22.780935 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:22.781540 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:22.781317 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:22.782061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:22.782045 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:05:24.874746 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.874706 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8"] Oct 30 06:05:24.880745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.880722 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.910339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.910293 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8"] Oct 30 06:05:24.911689 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.911661 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w"] Oct 30 06:05:24.915649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.915627 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:24.943915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.943892 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w"] Oct 30 06:05:24.958985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.958960 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.959108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959000 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:24.959108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959072 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.959227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959111 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.959227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959129 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20f0e3c-d224-467b-8246-a93ade40a4da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.959227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959152 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:24.959227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959198 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:24.959420 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959282 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.959420 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959326 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:24.959420 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959356 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.959420 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959373 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:24.959578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959428 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:24.959578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959508 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.959578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959528 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:24.959666 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959615 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whv8l\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-kube-api-access-whv8l\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.959666 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959646 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:24.959725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959681 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf654b-4fe7-448e-a7ba-05be6478a21d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:24.959725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:24.959699 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwrs\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-kube-api-access-8zwrs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.060707 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.060673 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whv8l\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-kube-api-access-whv8l\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.060855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.060715 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.060855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.060739 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf654b-4fe7-448e-a7ba-05be6478a21d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.060990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.060914 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwrs\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-kube-api-access-8zwrs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.060990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.060971 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.061109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.060990 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.061109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.061017 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.061109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.061040 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.061109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.061067 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20f0e3c-d224-467b-8246-a93ade40a4da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.061969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.061863 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf654b-4fe7-448e-a7ba-05be6478a21d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.061969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.061704 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.062289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.061096 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.062474 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.062267 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.062551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.062427 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.062617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.062586 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.062617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.062600 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.062812 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.062640 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.063417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.063390 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20f0e3c-d224-467b-8246-a93ade40a4da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.063677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.063484 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.063855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.063827 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.064101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.063892 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.064185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.064145 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.064797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.064205 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.064797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.064400 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.064797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.064576 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.065008 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.064686 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.065008 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.064985 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.065791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.065429 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.065907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.065873 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.066265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.066243 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.066387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.066364 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.066557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.066535 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.083864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.083844 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.084028 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.084004 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whv8l\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-kube-api-access-whv8l\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9jxv8\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.084136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.084122 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwrs\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-kube-api-access-8zwrs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.088180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.088162 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2q89w\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.194567 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.194542 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:25.229539 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.229510 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:25.508909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.508832 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8"] Oct 30 06:05:25.512495 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:05:25.512466 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20f0e3c_d224_467b_8246_a93ade40a4da.slice/crio-8dc5af640485004eb3ff0aed1fb152b43c8a5b6eac88e144494193dd2de11f82 WatchSource:0}: Error finding container 8dc5af640485004eb3ff0aed1fb152b43c8a5b6eac88e144494193dd2de11f82: Status 404 returned error can't find the container with id 8dc5af640485004eb3ff0aed1fb152b43c8a5b6eac88e144494193dd2de11f82 Oct 30 06:05:25.515495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.515464 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:05:25.515585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.515529 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:05:25.515585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.515560 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:05:25.552432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.552154 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w"] Oct 30 06:05:25.554257 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:05:25.554234 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf654b_4fe7_448e_a7ba_05be6478a21d.slice/crio-b1c02c7b0854a0c8bfd4fd3e3ed8c7a1769d8d5050babc02374ca1107bf023f0 WatchSource:0}: Error finding container b1c02c7b0854a0c8bfd4fd3e3ed8c7a1769d8d5050babc02374ca1107bf023f0: Status 404 returned error can't find the container with id b1c02c7b0854a0c8bfd4fd3e3ed8c7a1769d8d5050babc02374ca1107bf023f0 Oct 30 06:05:25.557194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.557160 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:05:25.557282 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.557259 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:05:25.557351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:25.557336 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:05:26.041508 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.041469 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" event={"ID":"b20f0e3c-d224-467b-8246-a93ade40a4da","Type":"ContainerStarted","Data":"a619164c0bc63cd10a713fd9ec2e0e36b911e728e993a32c5359e614bf97d0fc"} Oct 30 06:05:26.042013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.041536 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" event={"ID":"b20f0e3c-d224-467b-8246-a93ade40a4da","Type":"ContainerStarted","Data":"8dc5af640485004eb3ff0aed1fb152b43c8a5b6eac88e144494193dd2de11f82"} Oct 30 06:05:26.042903 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.042876 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" event={"ID":"2abf654b-4fe7-448e-a7ba-05be6478a21d","Type":"ContainerStarted","Data":"b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80"} Oct 30 06:05:26.043033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.042920 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" event={"ID":"2abf654b-4fe7-448e-a7ba-05be6478a21d","Type":"ContainerStarted","Data":"b1c02c7b0854a0c8bfd4fd3e3ed8c7a1769d8d5050babc02374ca1107bf023f0"} Oct 30 06:05:26.094141 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.093031 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podStartSLOduration=2.093014205 podStartE2EDuration="2.093014205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:26.091938448 +0000 UTC m=+734.898460197" watchObservedRunningTime="2026-10-30 06:05:26.093014205 +0000 UTC m=+734.899535953" Oct 30 06:05:26.138389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.138329 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podStartSLOduration=2.138287128 podStartE2EDuration="2.138287128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:26.137444612 +0000 UTC m=+734.943966361" watchObservedRunningTime="2026-10-30 06:05:26.138287128 +0000 UTC m=+734.944808877" Oct 30 06:05:26.200772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.195673 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:26.201442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.201068 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9jxv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 06:05:26.201442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.201135 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 06:05:26.229949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.229916 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:26.231171 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.231151 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2q89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:26.231244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:26.231195 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 06:05:27.196326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:27.195720 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9jxv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 06:05:27.196326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:27.195783 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 06:05:27.230886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:27.230856 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2q89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:27.231020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:27.230906 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 06:05:28.200325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:28.195433 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9jxv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 06:05:28.200325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:28.195495 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 06:05:28.230487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:28.230445 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2q89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:28.230621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:28.230497 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 06:05:29.196051 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:29.196017 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9jxv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 06:05:29.196475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:29.196444 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 06:05:29.230207 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:29.230175 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2q89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:29.230674 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:29.230229 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 06:05:30.195824 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:30.195785 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9jxv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 06:05:30.196127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:30.196097 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 06:05:30.230531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:30.230504 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2q89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:30.230863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:30.230547 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 06:05:31.197548 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:31.197512 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9jxv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 06:05:31.197815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:31.197788 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 06:05:31.230733 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:31.230705 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2q89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:31.231192 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:31.231163 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 06:05:32.195740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:32.195699 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9jxv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 06:05:32.195934 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:32.195766 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 06:05:32.230864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:32.230829 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2q89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:32.231185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:32.230890 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 06:05:33.713616 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:05:33.713577 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:05:33.794750 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:33.794715 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:33.795192 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:33.795164 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:33.796108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:33.796089 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:05:33.827281 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:33.827245 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:33.827683 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:33.827631 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:05:33.829796 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:05:33.829769 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:06:03.699318 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:03.699220 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:06:06.926633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.926604 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz"] Oct 30 06:06:06.930355 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.930337 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.950570 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.950541 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-istio-token\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.950678 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.950575 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b65921-33ab-4502-9780-bbf282f416d4-istio-podinfo\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.950678 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.950603 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b65921-33ab-4502-9780-bbf282f416d4-istiod-ca-cert\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.950753 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.950727 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-envoy\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.950793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.950762 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-certs\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.950829 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.950811 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-socket\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.950914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.950899 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxk8m\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-kube-api-access-rxk8m\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.950953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.950923 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-credential-socket\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.950953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.950938 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-data\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:06.953857 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.953837 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c45a2ff8-openshift-default-dockercfg-xzcv5\"" Oct 30 06:06:06.968531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:06.968509 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz"] Oct 30 06:06:07.052408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.052378 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-istio-token\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.052560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.052414 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b65921-33ab-4502-9780-bbf282f416d4-istio-podinfo\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.052560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.052455 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b65921-33ab-4502-9780-bbf282f416d4-istiod-ca-cert\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.052560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.052493 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-envoy\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.052716 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.052635 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-certs\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.052716 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.052675 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-socket\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.052816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.052714 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxk8m\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-kube-api-access-rxk8m\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.052816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.052741 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-credential-socket\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.052816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.052766 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-data\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.053066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.053041 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-socket\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.053190 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.053172 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-certs\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.053254 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.053210 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-credential-socket\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.053335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.053285 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-data\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.053390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.053335 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b65921-33ab-4502-9780-bbf282f416d4-istiod-ca-cert\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.054951 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.054931 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-envoy\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.055353 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.055337 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b65921-33ab-4502-9780-bbf282f416d4-istio-podinfo\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.083344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.083317 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxk8m\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-kube-api-access-rxk8m\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.086077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.086050 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-istio-token\") pod \"e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.096771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.096746 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf"] Oct 30 06:06:07.102836 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.102818 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.122705 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.122687 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9c94fbc2-openshift-default-dockercfg-mww7m\"" Oct 30 06:06:07.144838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.144811 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf"] Oct 30 06:06:07.153334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.153290 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-data\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.153452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.153383 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-podinfo\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.153452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.153402 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istiod-ca-cert\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.153558 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.153492 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq76f\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-kube-api-access-xq76f\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.153615 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.153596 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-envoy\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.153665 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.153618 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-token\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.153665 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.153637 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-certs\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.153665 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.153654 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-credential-socket\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.153812 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.153727 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-socket\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.246502 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.246481 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:07.254136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254110 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-envoy\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254142 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-token\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254160 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-certs\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254176 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-credential-socket\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254195 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-socket\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254223 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-data\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254257 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-podinfo\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254279 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istiod-ca-cert\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254344 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq76f\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-kube-api-access-xq76f\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254691 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254666 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-credential-socket\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.254902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254879 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-socket\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.255015 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.254945 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-certs\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.255173 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.255147 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-data\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.255565 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.255540 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istiod-ca-cert\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.256461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.256445 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-envoy\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.256967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.256950 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-podinfo\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.277069 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.277045 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-token\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.277286 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.277268 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq76f\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-kube-api-access-xq76f\") pod \"e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.417460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.417430 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:07.606093 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.606066 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz"] Oct 30 06:06:07.608576 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:07.608549 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b65921_33ab_4502_9780_bbf282f416d4.slice/crio-eeca493e3da802f0f3a21ba731e185e5e91514423a4dc0868adb4eca0e7ef342 WatchSource:0}: Error finding container eeca493e3da802f0f3a21ba731e185e5e91514423a4dc0868adb4eca0e7ef342: Status 404 returned error can't find the container with id eeca493e3da802f0f3a21ba731e185e5e91514423a4dc0868adb4eca0e7ef342 Oct 30 06:06:07.610745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.610712 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:07.610849 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.610783 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:07.610849 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.610813 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:07.723460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.723285 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf"] Oct 30 06:06:07.726291 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:07.726265 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3444fc_dd5a_401f_892c_5d2a4e5a7746.slice/crio-72601da643083274738ca7b10cfd44c9a45b226660c6dbfcc4e32f57a39fe38f WatchSource:0}: Error finding container 72601da643083274738ca7b10cfd44c9a45b226660c6dbfcc4e32f57a39fe38f: Status 404 returned error can't find the container with id 72601da643083274738ca7b10cfd44c9a45b226660c6dbfcc4e32f57a39fe38f Oct 30 06:06:07.729159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.729116 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:07.729255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.729194 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:07.729255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:07.729235 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:08.248049 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.248011 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" event={"ID":"fa3444fc-dd5a-401f-892c-5d2a4e5a7746","Type":"ContainerStarted","Data":"1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90"} Oct 30 06:06:08.248519 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.248065 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" event={"ID":"fa3444fc-dd5a-401f-892c-5d2a4e5a7746","Type":"ContainerStarted","Data":"72601da643083274738ca7b10cfd44c9a45b226660c6dbfcc4e32f57a39fe38f"} Oct 30 06:06:08.249671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.249645 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" event={"ID":"78b65921-33ab-4502-9780-bbf282f416d4","Type":"ContainerStarted","Data":"872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434"} Oct 30 06:06:08.249791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.249683 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" event={"ID":"78b65921-33ab-4502-9780-bbf282f416d4","Type":"ContainerStarted","Data":"eeca493e3da802f0f3a21ba731e185e5e91514423a4dc0868adb4eca0e7ef342"} Oct 30 06:06:08.295851 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.295806 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" podStartSLOduration=1.2957925829999999 podStartE2EDuration="1.295792583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:08.295094194 +0000 UTC m=+777.101615943" watchObservedRunningTime="2026-10-30 06:06:08.295792583 +0000 UTC m=+777.102314404" Oct 30 06:06:08.341522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.341480 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" podStartSLOduration=2.341468425 podStartE2EDuration="2.341468425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:08.339618746 +0000 UTC m=+777.146140494" watchObservedRunningTime="2026-10-30 06:06:08.341468425 +0000 UTC m=+777.147990174" Oct 30 06:06:08.417709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.417686 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:08.422626 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.422607 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:08.624631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.624550 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj"] Oct 30 06:06:08.628108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.628087 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.642862 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.642837 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-55140dc4-openshift-default-dockercfg-xj957\"" Oct 30 06:06:08.651933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.651896 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj"] Oct 30 06:06:08.667315 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.667275 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-token\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.667449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.667429 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-credential-socket\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.667491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.667481 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-socket\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.667600 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.667582 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2afa81d-0896-4648-b9d3-0ee402571a09-istiod-ca-cert\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.667647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.667635 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-certs\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.667748 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.667725 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-data\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.667780 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.667767 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-podinfo\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.667863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.667842 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-envoy\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.667923 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.667906 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf5tv\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-kube-api-access-hf5tv\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769179 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2afa81d-0896-4648-b9d3-0ee402571a09-istiod-ca-cert\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769230 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-certs\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769258 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-data\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769285 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-podinfo\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769354 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-envoy\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769400 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf5tv\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-kube-api-access-hf5tv\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769436 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-token\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769494 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-credential-socket\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769534 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-socket\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769694 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-certs\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769772 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-data\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.769951 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.769932 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-socket\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.770037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.770014 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-credential-socket\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.770081 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.770025 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2afa81d-0896-4648-b9d3-0ee402571a09-istiod-ca-cert\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.771735 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.771716 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-envoy\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.771855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.771838 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-podinfo\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.792485 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.792454 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-token\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.792632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.792614 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf5tv\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-kube-api-access-hf5tv\") pod \"e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:08.941247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:08.941217 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:09.246882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.246789 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:09.252092 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.252065 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:09.256108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.256023 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:09.256108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.256067 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:09.257858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.257828 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:09.257966 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.257885 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj"] Oct 30 06:06:09.258040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.257979 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:09.263511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.263472 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:09.263635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.263610 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:09.263738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:09.263712 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:10.262585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:10.262543 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" event={"ID":"e2afa81d-0896-4648-b9d3-0ee402571a09","Type":"ContainerStarted","Data":"f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e"} Oct 30 06:06:10.263014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:10.262629 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" event={"ID":"e2afa81d-0896-4648-b9d3-0ee402571a09","Type":"ContainerStarted","Data":"209ea95c3c7fb33a85b6e042db059f4ad37407e4fa721c91d1a5222c8fb9c5ee"} Oct 30 06:06:10.330610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:10.330561 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" podStartSLOduration=2.330545768 podStartE2EDuration="2.330545768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:10.329288419 +0000 UTC m=+779.135810182" watchObservedRunningTime="2026-10-30 06:06:10.330545768 +0000 UTC m=+779.137067516" Oct 30 06:06:10.941742 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:10.941708 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:10.947166 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:10.947145 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:11.267805 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.267729 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:11.268663 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.268645 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:11.579140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.579050 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs"] Oct 30 06:06:11.585427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.585401 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.597443 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.597421 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-maas-api-tls\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.597532 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.597457 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvkww\" (UniqueName: \"kubernetes.io/projected/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-kube-api-access-mvkww\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.597570 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.597546 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-openshift-service-ca\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.597603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.597595 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-metrics-certs\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.605461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.605439 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-798193ae-serving-cert\"" Oct 30 06:06:11.608016 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.607988 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs"] Oct 30 06:06:11.698657 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.698622 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-openshift-service-ca\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.698806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.698678 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-metrics-certs\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.698806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.698715 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-maas-api-tls\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.698806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.698733 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvkww\" (UniqueName: \"kubernetes.io/projected/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-kube-api-access-mvkww\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.699434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.699409 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-openshift-service-ca\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.701127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.701103 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-metrics-certs\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.701237 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.701214 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-maas-api-tls\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.721229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.721206 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvkww\" (UniqueName: \"kubernetes.io/projected/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-kube-api-access-mvkww\") pod \"maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:11.898230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:11.898148 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:12.228094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:12.227815 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs"] Oct 30 06:06:12.234561 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:12.234535 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbc2e64_14e6_4b45_8ee7_05c1ff65c3f5.slice/crio-8ff80756fdf45c1474d8f51c6c93eea75595e4b3a81dbda3f58e29ae75f42681 WatchSource:0}: Error finding container 8ff80756fdf45c1474d8f51c6c93eea75595e4b3a81dbda3f58e29ae75f42681: Status 404 returned error can't find the container with id 8ff80756fdf45c1474d8f51c6c93eea75595e4b3a81dbda3f58e29ae75f42681 Oct 30 06:06:12.273258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:12.273225 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" event={"ID":"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5","Type":"ContainerStarted","Data":"8ff80756fdf45c1474d8f51c6c93eea75595e4b3a81dbda3f58e29ae75f42681"} Oct 30 06:06:13.279256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.279216 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" event={"ID":"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5","Type":"ContainerStarted","Data":"8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3"} Oct 30 06:06:13.279747 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.279287 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:13.326125 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.326075 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" podStartSLOduration=1.83714538 podStartE2EDuration="2.326061999s" totalImagesPullingTime="488.916619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:13.323756796 +0000 UTC m=+782.130278545" watchObservedRunningTime="2026-10-30 06:06:13.326061999 +0000 UTC m=+782.132583746" Oct 30 06:06:13.705578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.705553 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb"] Oct 30 06:06:13.709184 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.709168 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.737253 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.736402 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c45a2ff8-serving-cert\"" Oct 30 06:06:13.737958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.737935 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb"] Oct 30 06:06:13.823878 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.823837 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-maas-api-tls\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.823878 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.823873 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4fl\" (UniqueName: \"kubernetes.io/projected/9c6c4792-8255-4bcb-abb7-6c497ca4a661-kube-api-access-zf4fl\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.824071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.824015 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-metrics-certs\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.824123 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.824106 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6c4792-8255-4bcb-abb7-6c497ca4a661-openshift-service-ca\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.925196 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.925166 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6c4792-8255-4bcb-abb7-6c497ca4a661-openshift-service-ca\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.925377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.925227 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-maas-api-tls\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.925377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.925255 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4fl\" (UniqueName: \"kubernetes.io/projected/9c6c4792-8255-4bcb-abb7-6c497ca4a661-kube-api-access-zf4fl\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.925377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.925353 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-metrics-certs\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.925850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.925827 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6c4792-8255-4bcb-abb7-6c497ca4a661-openshift-service-ca\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.927532 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.927507 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-maas-api-tls\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.927706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.927686 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-metrics-certs\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:13.952912 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:13.952884 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4fl\" (UniqueName: \"kubernetes.io/projected/9c6c4792-8255-4bcb-abb7-6c497ca4a661-kube-api-access-zf4fl\") pod \"maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:14.032545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:14.021759 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:14.350571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:14.350545 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb"] Oct 30 06:06:14.352283 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:14.352255 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6c4792_8255_4bcb_abb7_6c497ca4a661.slice/crio-e5c0e35bcf5d98a195eac236957c1f82c946c603ff162c179936023820bdbf62 WatchSource:0}: Error finding container e5c0e35bcf5d98a195eac236957c1f82c946c603ff162c179936023820bdbf62: Status 404 returned error can't find the container with id e5c0e35bcf5d98a195eac236957c1f82c946c603ff162c179936023820bdbf62 Oct 30 06:06:15.291515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:15.291466 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" event={"ID":"9c6c4792-8255-4bcb-abb7-6c497ca4a661","Type":"ContainerStarted","Data":"3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1"} Oct 30 06:06:15.291698 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:15.291532 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" event={"ID":"9c6c4792-8255-4bcb-abb7-6c497ca4a661","Type":"ContainerStarted","Data":"e5c0e35bcf5d98a195eac236957c1f82c946c603ff162c179936023820bdbf62"} Oct 30 06:06:15.291698 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:15.291667 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:16.122406 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.122344 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" podStartSLOduration=2.810758454 podStartE2EDuration="3.122327928s" totalImagesPullingTime="311.569474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:15.341888744 +0000 UTC m=+784.148410493" watchObservedRunningTime="2026-10-30 06:06:16.122327928 +0000 UTC m=+784.928849673" Oct 30 06:06:16.122893 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.122545 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c"] Oct 30 06:06:16.126220 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.126195 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.155461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.155431 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c"] Oct 30 06:06:16.229346 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.229296 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9c94fbc2-serving-cert\"" Oct 30 06:06:16.251022 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.250993 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b947f444-7f82-4ccb-b81e-691633617e78-openshift-service-ca\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.251129 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.251042 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748r6\" (UniqueName: \"kubernetes.io/projected/b947f444-7f82-4ccb-b81e-691633617e78-kube-api-access-748r6\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.251184 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.251161 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-metrics-certs\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.251341 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.251292 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-maas-api-tls\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.352368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.352293 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-maas-api-tls\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.352608 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.352448 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b947f444-7f82-4ccb-b81e-691633617e78-openshift-service-ca\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.352608 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.352505 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748r6\" (UniqueName: \"kubernetes.io/projected/b947f444-7f82-4ccb-b81e-691633617e78-kube-api-access-748r6\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.352608 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.352558 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-metrics-certs\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.353711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.353680 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b947f444-7f82-4ccb-b81e-691633617e78-openshift-service-ca\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.355387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.355358 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-maas-api-tls\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.355658 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.355626 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-metrics-certs\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.394268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.394210 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-748r6\" (UniqueName: \"kubernetes.io/projected/b947f444-7f82-4ccb-b81e-691633617e78-kube-api-access-748r6\") pod \"maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.440355 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.440327 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:16.745205 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:16.745178 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c"] Oct 30 06:06:16.746750 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:16.746722 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb947f444_7f82_4ccb_b81e_691633617e78.slice/crio-e9b37234a83fdedaab654de1a0dee5bad5bc5972a9b1cd05d44cc602251ca6e6 WatchSource:0}: Error finding container e9b37234a83fdedaab654de1a0dee5bad5bc5972a9b1cd05d44cc602251ca6e6: Status 404 returned error can't find the container with id e9b37234a83fdedaab654de1a0dee5bad5bc5972a9b1cd05d44cc602251ca6e6 Oct 30 06:06:17.304445 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:17.304409 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" event={"ID":"b947f444-7f82-4ccb-b81e-691633617e78","Type":"ContainerStarted","Data":"84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82"} Oct 30 06:06:17.304839 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:17.304462 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" event={"ID":"b947f444-7f82-4ccb-b81e-691633617e78","Type":"ContainerStarted","Data":"e9b37234a83fdedaab654de1a0dee5bad5bc5972a9b1cd05d44cc602251ca6e6"} Oct 30 06:06:17.304839 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:17.304544 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:17.355942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:17.355895 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" podStartSLOduration=0.996249232 podStartE2EDuration="1.355882093s" totalImagesPullingTime="359.632861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:17.355422765 +0000 UTC m=+786.161944513" watchObservedRunningTime="2026-10-30 06:06:17.355882093 +0000 UTC m=+786.162403840" Oct 30 06:06:19.290211 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:19.290181 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:20.390213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.390183 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf"] Oct 30 06:06:20.394009 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.393989 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.432655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.432628 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-55140dc4-serving-cert\"" Oct 30 06:06:20.450213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.450189 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf"] Oct 30 06:06:20.496276 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.496244 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-maas-api-tls\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.496417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.496284 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-metrics-certs\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.496495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.496441 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52607b4-c7bf-443a-a2f2-4989d4528e49-openshift-service-ca\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.496603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.496588 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prpkz\" (UniqueName: \"kubernetes.io/projected/e52607b4-c7bf-443a-a2f2-4989d4528e49-kube-api-access-prpkz\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.597430 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.597400 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpkz\" (UniqueName: \"kubernetes.io/projected/e52607b4-c7bf-443a-a2f2-4989d4528e49-kube-api-access-prpkz\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.597560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.597449 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-maas-api-tls\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.597604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.597556 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-metrics-certs\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.597654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.597638 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52607b4-c7bf-443a-a2f2-4989d4528e49-openshift-service-ca\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.598255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.598228 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52607b4-c7bf-443a-a2f2-4989d4528e49-openshift-service-ca\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.599797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.599772 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-maas-api-tls\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.599876 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.599840 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-metrics-certs\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.620276 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.620253 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpkz\" (UniqueName: \"kubernetes.io/projected/e52607b4-c7bf-443a-a2f2-4989d4528e49-kube-api-access-prpkz\") pod \"maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:20.709430 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:20.709408 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:21.011573 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:21.011530 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf"] Oct 30 06:06:21.013683 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:21.013658 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52607b4_c7bf_443a_a2f2_4989d4528e49.slice/crio-5de0ef838fca0587ccc76d10ac6b4bc0fe7878081ebb0e17e4de753aebe4e674 WatchSource:0}: Error finding container 5de0ef838fca0587ccc76d10ac6b4bc0fe7878081ebb0e17e4de753aebe4e674: Status 404 returned error can't find the container with id 5de0ef838fca0587ccc76d10ac6b4bc0fe7878081ebb0e17e4de753aebe4e674 Oct 30 06:06:21.303526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:21.303456 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:21.325208 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:21.325172 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" event={"ID":"e52607b4-c7bf-443a-a2f2-4989d4528e49","Type":"ContainerStarted","Data":"5de0ef838fca0587ccc76d10ac6b4bc0fe7878081ebb0e17e4de753aebe4e674"} Oct 30 06:06:22.331475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:22.331436 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" event={"ID":"e52607b4-c7bf-443a-a2f2-4989d4528e49","Type":"ContainerStarted","Data":"74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09"} Oct 30 06:06:22.331978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:22.331576 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:23.317118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:23.317087 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:23.379333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:23.378482 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" podStartSLOduration=2.886814555 podStartE2EDuration="3.378463432s" totalImagesPullingTime="491.648877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:22.380475109 +0000 UTC m=+791.186996873" watchObservedRunningTime="2026-10-30 06:06:23.378463432 +0000 UTC m=+792.184985180" Oct 30 06:06:28.352504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:28.352457 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:30.477493 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.477452 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl"] Oct 30 06:06:30.481586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.481565 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.507437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.507411 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3627d700-serving-cert\"" Oct 30 06:06:30.519460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.519436 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl"] Oct 30 06:06:30.598105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.598068 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-metrics-certs\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.598276 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.598138 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlgd\" (UniqueName: \"kubernetes.io/projected/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-kube-api-access-qmlgd\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.598276 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.598218 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-openshift-service-ca\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.598402 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.598372 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.699483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.699443 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.699685 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.699555 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-metrics-certs\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.699685 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:30.699632 4431 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-serving-cert: secret "maas-api-e2e-derive-3627d700-serving-cert" not found Oct 30 06:06:30.699995 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:30.699723 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls podName:8ab5cc92-9099-48d0-913b-c8b4ff7160ff nodeName:}" failed. No retries permitted until 2026-10-30 06:06:31.199701121 +0000 UTC m=+800.006222871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls") pod "maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" (UID: "8ab5cc92-9099-48d0-913b-c8b4ff7160ff") : secret "maas-api-e2e-derive-3627d700-serving-cert" not found Oct 30 06:06:30.699995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.699635 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlgd\" (UniqueName: \"kubernetes.io/projected/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-kube-api-access-qmlgd\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.699995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.699916 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-openshift-service-ca\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.700870 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.700842 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-openshift-service-ca\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.702929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.702897 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-metrics-certs\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:30.745090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:30.744994 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlgd\" (UniqueName: \"kubernetes.io/projected/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-kube-api-access-qmlgd\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:31.206315 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.206268 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:31.208893 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.208863 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls\") pod \"maas-api-e2e-derive-3627d700-7945bfc496-fqbjl\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:31.399333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.399220 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:31.414374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.414347 4431 generic.go:356] "Generic (PLEG): container finished" podID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerID="1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90" exitCode=0 Oct 30 06:06:31.414499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.414419 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" event={"ID":"fa3444fc-dd5a-401f-892c-5d2a4e5a7746","Type":"ContainerDied","Data":"1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90"} Oct 30 06:06:31.414868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.414837 4431 scope.go:122] "RemoveContainer" containerID="1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90" Oct 30 06:06:31.416542 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.416506 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:31.416625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.416606 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:31.416668 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.416654 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:31.417825 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.417801 4431 generic.go:356] "Generic (PLEG): container finished" podID="78b65921-33ab-4502-9780-bbf282f416d4" containerID="872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434" exitCode=0 Oct 30 06:06:31.417940 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.417848 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" event={"ID":"78b65921-33ab-4502-9780-bbf282f416d4","Type":"ContainerDied","Data":"872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434"} Oct 30 06:06:31.418195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.418178 4431 scope.go:122] "RemoveContainer" containerID="872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434" Oct 30 06:06:31.419583 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.419540 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:31.419650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.419631 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:31.419698 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.419684 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:31.734816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:31.724879 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl"] Oct 30 06:06:32.426518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:32.426437 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" event={"ID":"fa3444fc-dd5a-401f-892c-5d2a4e5a7746","Type":"ContainerStarted","Data":"27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e"} Oct 30 06:06:32.429147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:32.429123 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" event={"ID":"78b65921-33ab-4502-9780-bbf282f416d4","Type":"ContainerStarted","Data":"4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b"} Oct 30 06:06:32.430528 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:32.430502 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" event={"ID":"8ab5cc92-9099-48d0-913b-c8b4ff7160ff","Type":"ContainerStarted","Data":"fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027"} Oct 30 06:06:32.430611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:32.430539 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" event={"ID":"8ab5cc92-9099-48d0-913b-c8b4ff7160ff","Type":"ContainerStarted","Data":"d4a8d08d1103d52191377322e44f919c29a361d9db644aa6b97b400681ab7e71"} Oct 30 06:06:32.430611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:32.430575 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:32.587343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:32.585742 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" podStartSLOduration=2.317259929 podStartE2EDuration="2.585722438s" totalImagesPullingTime="268.462509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:32.583861652 +0000 UTC m=+801.390383401" watchObservedRunningTime="2026-10-30 06:06:32.585722438 +0000 UTC m=+801.392244187" Oct 30 06:06:33.247715 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:33.247678 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:33.248095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:33.247795 4431 patch_prober.go:29] interesting pod/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:06:33.248095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:33.247845 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 06:06:33.420345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:33.417839 4431 patch_prober.go:29] interesting pod/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:06:33.420345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:33.417897 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 06:06:33.420345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:33.418170 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:33.741533 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:33.741497 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:06:34.247745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:34.247717 4431 patch_prober.go:29] interesting pod/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:06:34.248186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:34.248158 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 06:06:34.419345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:34.418688 4431 patch_prober.go:29] interesting pod/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:06:34.419345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:34.418751 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 06:06:35.246975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:35.246935 4431 patch_prober.go:29] interesting pod/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:06:35.247141 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:35.246993 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 06:06:35.418036 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:35.418002 4431 patch_prober.go:29] interesting pod/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:06:35.418499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:35.418468 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 06:06:36.247464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.247428 4431 patch_prober.go:29] interesting pod/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:06:36.247722 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.247523 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 06:06:36.418202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.418174 4431 patch_prober.go:29] interesting pod/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:06:36.418655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.418394 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 06:06:36.453984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.453954 4431 generic.go:356] "Generic (PLEG): container finished" podID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerID="f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e" exitCode=0 Oct 30 06:06:36.454142 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.454033 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" event={"ID":"e2afa81d-0896-4648-b9d3-0ee402571a09","Type":"ContainerDied","Data":"f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e"} Oct 30 06:06:36.454504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.454486 4431 scope.go:122] "RemoveContainer" containerID="f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e" Oct 30 06:06:36.455991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.455967 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:36.456057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.456031 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:36.456096 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:36.456067 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:37.248000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:37.247962 4431 patch_prober.go:29] interesting pod/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:06:37.248175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:37.248017 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 06:06:37.423329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:37.422744 4431 patch_prober.go:29] interesting pod/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:06:37.423329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:37.422809 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 06:06:37.465825 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:37.465791 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" event={"ID":"e2afa81d-0896-4648-b9d3-0ee402571a09","Type":"ContainerStarted","Data":"87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab"} Oct 30 06:06:37.941634 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:37.941597 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:37.942765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:37.942739 4431 patch_prober.go:29] interesting pod/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 06:06:37.942845 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:37.942788 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 06:06:38.251192 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:38.251113 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:38.251426 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:38.251408 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:38.252156 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:38.252139 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:38.422749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:38.422719 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:38.423047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:38.423028 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:38.423809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:38.423789 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:38.441652 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:38.441634 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:06:38.942501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:38.942466 4431 patch_prober.go:29] interesting pod/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 06:06:38.942673 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:38.942516 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 06:06:39.941818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:39.941787 4431 patch_prober.go:29] interesting pod/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 06:06:39.942255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:39.941843 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 06:06:40.942073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:40.942038 4431 patch_prober.go:29] interesting pod/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 06:06:40.942564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:40.942102 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 06:06:41.284383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:41.279389 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:41.942135 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:41.942095 4431 patch_prober.go:29] interesting pod/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 06:06:41.942523 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:41.942169 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 06:06:42.947171 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:42.947133 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:42.948241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:42.948217 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:46.259006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.258973 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs"] Oct 30 06:06:46.259510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.259243 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" podUID="bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" containerName="maas-api" containerID="cri-o://8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3" gracePeriod=30 Oct 30 06:06:46.523057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.523005 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:46.529913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.529886 4431 generic.go:356] "Generic (PLEG): container finished" podID="bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" containerID="8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3" exitCode=0 Oct 30 06:06:46.530034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.529934 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" Oct 30 06:06:46.530034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.529966 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" event={"ID":"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5","Type":"ContainerDied","Data":"8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3"} Oct 30 06:06:46.530202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.530030 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs" event={"ID":"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5","Type":"ContainerDied","Data":"8ff80756fdf45c1474d8f51c6c93eea75595e4b3a81dbda3f58e29ae75f42681"} Oct 30 06:06:46.530202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.530064 4431 scope.go:122] "RemoveContainer" containerID="8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3" Oct 30 06:06:46.538359 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.538341 4431 scope.go:122] "RemoveContainer" containerID="8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3" Oct 30 06:06:46.538615 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:46.538594 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3\": container with ID starting with 8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3 not found: ID does not exist" containerID="8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3" Oct 30 06:06:46.538666 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.538624 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3"} err="failed to get container status \"8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3\": rpc error: code = NotFound desc = could not find container \"8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3\": container with ID starting with 8e213c8949ed23ec5a13b48fc5bf4b0a68904f627ca2f0bf7f33e10f533562e3 not found: ID does not exist" Oct 30 06:06:46.551866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.551836 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-openshift-service-ca\") pod \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " Oct 30 06:06:46.551951 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.551917 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-kube-api-access-mvkww\" (UniqueName: \"kubernetes.io/projected/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-kube-api-access-mvkww\") pod \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " Oct 30 06:06:46.552001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.551990 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-metrics-certs\") pod \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " Oct 30 06:06:46.552053 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.552040 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-maas-api-tls\") pod \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\" (UID: \"bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5\") " Oct 30 06:06:46.552243 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.552216 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-openshift-service-ca" pod "bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" (UID: "bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:46.552399 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.552381 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:46.554263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.554242 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-metrics-certs" pod "bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" (UID: "bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:46.554366 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.554289 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-kube-api-access-mvkww" pod "bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" (UID: "bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5"). InnerVolumeSpecName "kube-api-access-mvkww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:46.554418 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.554366 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-maas-api-tls" pod "bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" (UID: "bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:46.653817 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.653790 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:46.653817 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.653813 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:46.653817 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.653822 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvkww\" (UniqueName: \"kubernetes.io/projected/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5-kube-api-access-mvkww\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:46.675718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.675688 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb"] Oct 30 06:06:46.675994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.675936 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" podUID="9c6c4792-8255-4bcb-abb7-6c497ca4a661" containerName="maas-api" containerID="cri-o://3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1" gracePeriod=30 Oct 30 06:06:46.876953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.876916 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs"] Oct 30 06:06:46.892925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.892890 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-798193ae-f7fb896fb-8d6fs"] Oct 30 06:06:46.917943 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.917922 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:46.956630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.956607 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-metrics-certs\") pod \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " Oct 30 06:06:46.956771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.956671 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6c4792-8255-4bcb-abb7-6c497ca4a661-kube-api-access-zf4fl\" (UniqueName: \"kubernetes.io/projected/9c6c4792-8255-4bcb-abb7-6c497ca4a661-kube-api-access-zf4fl\") pod \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " Oct 30 06:06:46.956833 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.956814 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-maas-api-tls\") pod \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " Oct 30 06:06:46.956884 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.956867 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6c4792-8255-4bcb-abb7-6c497ca4a661-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6c4792-8255-4bcb-abb7-6c497ca4a661-openshift-service-ca\") pod \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\" (UID: \"9c6c4792-8255-4bcb-abb7-6c497ca4a661\") " Oct 30 06:06:46.957265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.957237 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6c4792-8255-4bcb-abb7-6c497ca4a661-openshift-service-ca" pod "9c6c4792-8255-4bcb-abb7-6c497ca4a661" (UID: "9c6c4792-8255-4bcb-abb7-6c497ca4a661"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:46.958771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.958741 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-metrics-certs" pod "9c6c4792-8255-4bcb-abb7-6c497ca4a661" (UID: "9c6c4792-8255-4bcb-abb7-6c497ca4a661"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:46.958872 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.958816 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-maas-api-tls" pod "9c6c4792-8255-4bcb-abb7-6c497ca4a661" (UID: "9c6c4792-8255-4bcb-abb7-6c497ca4a661"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:46.958960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:46.958943 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6c4792-8255-4bcb-abb7-6c497ca4a661-kube-api-access-zf4fl" pod "9c6c4792-8255-4bcb-abb7-6c497ca4a661" (UID: "9c6c4792-8255-4bcb-abb7-6c497ca4a661"). InnerVolumeSpecName "kube-api-access-zf4fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:47.058423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.058329 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.058423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.058383 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6c4792-8255-4bcb-abb7-6c497ca4a661-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.058423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.058399 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6c4792-8255-4bcb-abb7-6c497ca4a661-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.058423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.058413 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf4fl\" (UniqueName: \"kubernetes.io/projected/9c6c4792-8255-4bcb-abb7-6c497ca4a661-kube-api-access-zf4fl\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.145181 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.145145 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c"] Oct 30 06:06:47.145659 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.145474 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" podUID="b947f444-7f82-4ccb-b81e-691633617e78" containerName="maas-api" containerID="cri-o://84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82" gracePeriod=30 Oct 30 06:06:47.337165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.337136 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" path="/var/lib/kubelet/pods/bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5/volumes" Oct 30 06:06:47.399014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.398993 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:47.462913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.462887 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-maas-api-tls\") pod \"b947f444-7f82-4ccb-b81e-691633617e78\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " Oct 30 06:06:47.463053 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.462925 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b947f444-7f82-4ccb-b81e-691633617e78-kube-api-access-748r6\" (UniqueName: \"kubernetes.io/projected/b947f444-7f82-4ccb-b81e-691633617e78-kube-api-access-748r6\") pod \"b947f444-7f82-4ccb-b81e-691633617e78\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " Oct 30 06:06:47.463053 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.462954 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-metrics-certs\") pod \"b947f444-7f82-4ccb-b81e-691633617e78\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " Oct 30 06:06:47.463053 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.463045 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b947f444-7f82-4ccb-b81e-691633617e78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b947f444-7f82-4ccb-b81e-691633617e78-openshift-service-ca\") pod \"b947f444-7f82-4ccb-b81e-691633617e78\" (UID: \"b947f444-7f82-4ccb-b81e-691633617e78\") " Oct 30 06:06:47.463662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.463634 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b947f444-7f82-4ccb-b81e-691633617e78-openshift-service-ca" pod "b947f444-7f82-4ccb-b81e-691633617e78" (UID: "b947f444-7f82-4ccb-b81e-691633617e78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:47.465067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.465044 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-maas-api-tls" pod "b947f444-7f82-4ccb-b81e-691633617e78" (UID: "b947f444-7f82-4ccb-b81e-691633617e78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:47.465365 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.465337 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-metrics-certs" pod "b947f444-7f82-4ccb-b81e-691633617e78" (UID: "b947f444-7f82-4ccb-b81e-691633617e78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:47.465474 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.465401 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b947f444-7f82-4ccb-b81e-691633617e78-kube-api-access-748r6" pod "b947f444-7f82-4ccb-b81e-691633617e78" (UID: "b947f444-7f82-4ccb-b81e-691633617e78"). InnerVolumeSpecName "kube-api-access-748r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:47.538065 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.538032 4431 generic.go:356] "Generic (PLEG): container finished" podID="9c6c4792-8255-4bcb-abb7-6c497ca4a661" containerID="3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1" exitCode=0 Oct 30 06:06:47.538213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.538093 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" Oct 30 06:06:47.538213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.538109 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" event={"ID":"9c6c4792-8255-4bcb-abb7-6c497ca4a661","Type":"ContainerDied","Data":"3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1"} Oct 30 06:06:47.538213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.538176 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb" event={"ID":"9c6c4792-8255-4bcb-abb7-6c497ca4a661","Type":"ContainerDied","Data":"e5c0e35bcf5d98a195eac236957c1f82c946c603ff162c179936023820bdbf62"} Oct 30 06:06:47.538374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.538222 4431 scope.go:122] "RemoveContainer" containerID="3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1" Oct 30 06:06:47.539711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.539685 4431 generic.go:356] "Generic (PLEG): container finished" podID="b947f444-7f82-4ccb-b81e-691633617e78" containerID="84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82" exitCode=0 Oct 30 06:06:47.539816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.539770 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" event={"ID":"b947f444-7f82-4ccb-b81e-691633617e78","Type":"ContainerDied","Data":"84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82"} Oct 30 06:06:47.539816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.539797 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" Oct 30 06:06:47.539932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.539830 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c" event={"ID":"b947f444-7f82-4ccb-b81e-691633617e78","Type":"ContainerDied","Data":"e9b37234a83fdedaab654de1a0dee5bad5bc5972a9b1cd05d44cc602251ca6e6"} Oct 30 06:06:47.547245 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.547227 4431 scope.go:122] "RemoveContainer" containerID="3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1" Oct 30 06:06:47.547512 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:47.547492 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1\": container with ID starting with 3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1 not found: ID does not exist" containerID="3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1" Oct 30 06:06:47.547608 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.547524 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1"} err="failed to get container status \"3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1\": rpc error: code = NotFound desc = could not find container \"3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1\": container with ID starting with 3a3b2d7ba56c4682437763d6e2d5631af860bdf088ad7e647fab37ca456260b1 not found: ID does not exist" Oct 30 06:06:47.547608 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.547547 4431 scope.go:122] "RemoveContainer" containerID="84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82" Oct 30 06:06:47.555790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.555764 4431 scope.go:122] "RemoveContainer" containerID="84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82" Oct 30 06:06:47.556065 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:47.556044 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82\": container with ID starting with 84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82 not found: ID does not exist" containerID="84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82" Oct 30 06:06:47.556143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.556101 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82"} err="failed to get container status \"84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82\": rpc error: code = NotFound desc = could not find container \"84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82\": container with ID starting with 84055c731657cd75406774b28f898a511dc7fdec262237237ab926f22d99be82 not found: ID does not exist" Oct 30 06:06:47.564585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.564564 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-748r6\" (UniqueName: \"kubernetes.io/projected/b947f444-7f82-4ccb-b81e-691633617e78-kube-api-access-748r6\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.564585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.564584 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.564700 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.564593 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b947f444-7f82-4ccb-b81e-691633617e78-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.564700 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.564602 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b947f444-7f82-4ccb-b81e-691633617e78-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.592230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.591563 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb"] Oct 30 06:06:47.601218 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.601165 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c45a2ff8-75c54464f5-hk9nb"] Oct 30 06:06:47.631603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.631572 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz"] Oct 30 06:06:47.631847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.631821 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" containerID="cri-o://4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b" gracePeriod=30 Oct 30 06:06:47.645779 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.645752 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c"] Oct 30 06:06:47.650859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:47.650829 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c94fbc2-6bcf7dcb96-gdt5c"] Oct 30 06:06:49.336488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:49.336454 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6c4792-8255-4bcb-abb7-6c497ca4a661" path="/var/lib/kubelet/pods/9c6c4792-8255-4bcb-abb7-6c497ca4a661/volumes" Oct 30 06:06:49.336902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:49.336853 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b947f444-7f82-4ccb-b81e-691633617e78" path="/var/lib/kubelet/pods/b947f444-7f82-4ccb-b81e-691633617e78/volumes" Oct 30 06:06:49.938705 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:49.938668 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf"] Oct 30 06:06:49.938929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:49.938904 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" containerID="cri-o://27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e" gracePeriod=30 Oct 30 06:06:51.823961 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:51.823927 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf"] Oct 30 06:06:51.824395 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:51.824172 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" podUID="e52607b4-c7bf-443a-a2f2-4989d4528e49" containerName="maas-api" containerID="cri-o://74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09" gracePeriod=30 Oct 30 06:06:52.072894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.072861 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:52.106035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.105949 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-maas-api-tls\") pod \"e52607b4-c7bf-443a-a2f2-4989d4528e49\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " Oct 30 06:06:52.106035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.105992 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-metrics-certs\") pod \"e52607b4-c7bf-443a-a2f2-4989d4528e49\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " Oct 30 06:06:52.106035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.106021 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52607b4-c7bf-443a-a2f2-4989d4528e49-kube-api-access-prpkz\" (UniqueName: \"kubernetes.io/projected/e52607b4-c7bf-443a-a2f2-4989d4528e49-kube-api-access-prpkz\") pod \"e52607b4-c7bf-443a-a2f2-4989d4528e49\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " Oct 30 06:06:52.106285 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.106074 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52607b4-c7bf-443a-a2f2-4989d4528e49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52607b4-c7bf-443a-a2f2-4989d4528e49-openshift-service-ca\") pod \"e52607b4-c7bf-443a-a2f2-4989d4528e49\" (UID: \"e52607b4-c7bf-443a-a2f2-4989d4528e49\") " Oct 30 06:06:52.106726 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.106684 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52607b4-c7bf-443a-a2f2-4989d4528e49-openshift-service-ca" pod "e52607b4-c7bf-443a-a2f2-4989d4528e49" (UID: "e52607b4-c7bf-443a-a2f2-4989d4528e49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:52.108412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.108385 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52607b4-c7bf-443a-a2f2-4989d4528e49-kube-api-access-prpkz" pod "e52607b4-c7bf-443a-a2f2-4989d4528e49" (UID: "e52607b4-c7bf-443a-a2f2-4989d4528e49"). InnerVolumeSpecName "kube-api-access-prpkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:52.108669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.108646 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-metrics-certs" pod "e52607b4-c7bf-443a-a2f2-4989d4528e49" (UID: "e52607b4-c7bf-443a-a2f2-4989d4528e49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:52.108754 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.108670 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-maas-api-tls" pod "e52607b4-c7bf-443a-a2f2-4989d4528e49" (UID: "e52607b4-c7bf-443a-a2f2-4989d4528e49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:52.206959 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.206923 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.206959 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.206953 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e52607b4-c7bf-443a-a2f2-4989d4528e49-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.206959 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.206964 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prpkz\" (UniqueName: \"kubernetes.io/projected/e52607b4-c7bf-443a-a2f2-4989d4528e49-kube-api-access-prpkz\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.207187 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.206976 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52607b4-c7bf-443a-a2f2-4989d4528e49-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:52.569992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.569961 4431 generic.go:356] "Generic (PLEG): container finished" podID="e52607b4-c7bf-443a-a2f2-4989d4528e49" containerID="74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09" exitCode=0 Oct 30 06:06:52.570174 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.570021 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" Oct 30 06:06:52.570174 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.570040 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" event={"ID":"e52607b4-c7bf-443a-a2f2-4989d4528e49","Type":"ContainerDied","Data":"74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09"} Oct 30 06:06:52.570174 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.570091 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf" event={"ID":"e52607b4-c7bf-443a-a2f2-4989d4528e49","Type":"ContainerDied","Data":"5de0ef838fca0587ccc76d10ac6b4bc0fe7878081ebb0e17e4de753aebe4e674"} Oct 30 06:06:52.570174 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.570112 4431 scope.go:122] "RemoveContainer" containerID="74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09" Oct 30 06:06:52.579757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.579741 4431 scope.go:122] "RemoveContainer" containerID="74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09" Oct 30 06:06:52.580073 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:52.580055 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09\": container with ID starting with 74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09 not found: ID does not exist" containerID="74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09" Oct 30 06:06:52.580139 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.580080 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09"} err="failed to get container status \"74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09\": rpc error: code = NotFound desc = could not find container \"74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09\": container with ID starting with 74fe1967408fe012f3d1daac46cc8e414df8f1eb27e1d394cb5c9241c5e65f09 not found: ID does not exist" Oct 30 06:06:52.629282 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.629256 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf"] Oct 30 06:06:52.639760 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.639737 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55140dc4-58f8fddfb8-rgftf"] Oct 30 06:06:52.902136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:52.902116 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:53.016032 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016005 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b65921-33ab-4502-9780-bbf282f416d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b65921-33ab-4502-9780-bbf282f416d4-istiod-ca-cert\") pod \"78b65921-33ab-4502-9780-bbf282f416d4\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " Oct 30 06:06:53.016185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016043 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-data\") pod \"78b65921-33ab-4502-9780-bbf282f416d4\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " Oct 30 06:06:53.016185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016065 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78b65921-33ab-4502-9780-bbf282f416d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b65921-33ab-4502-9780-bbf282f416d4-istio-podinfo\") pod \"78b65921-33ab-4502-9780-bbf282f416d4\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " Oct 30 06:06:53.016185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016134 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-istio-token\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-istio-token\") pod \"78b65921-33ab-4502-9780-bbf282f416d4\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " Oct 30 06:06:53.016185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016158 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-envoy\") pod \"78b65921-33ab-4502-9780-bbf282f416d4\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " Oct 30 06:06:53.016185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016183 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-kube-api-access-rxk8m\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-kube-api-access-rxk8m\") pod \"78b65921-33ab-4502-9780-bbf282f416d4\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " Oct 30 06:06:53.016432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016241 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-socket\") pod \"78b65921-33ab-4502-9780-bbf282f416d4\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " Oct 30 06:06:53.016432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016281 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-credential-socket\") pod \"78b65921-33ab-4502-9780-bbf282f416d4\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " Oct 30 06:06:53.016432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016333 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-certs\") pod \"78b65921-33ab-4502-9780-bbf282f416d4\" (UID: \"78b65921-33ab-4502-9780-bbf282f416d4\") " Oct 30 06:06:53.016432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016332 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-data" pod "78b65921-33ab-4502-9780-bbf282f416d4" (UID: "78b65921-33ab-4502-9780-bbf282f416d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:53.016647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016471 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b65921-33ab-4502-9780-bbf282f416d4-istiod-ca-cert" pod "78b65921-33ab-4502-9780-bbf282f416d4" (UID: "78b65921-33ab-4502-9780-bbf282f416d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:53.016704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016652 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-socket" pod "78b65921-33ab-4502-9780-bbf282f416d4" (UID: "78b65921-33ab-4502-9780-bbf282f416d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:53.016704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016681 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-certs" pod "78b65921-33ab-4502-9780-bbf282f416d4" (UID: "78b65921-33ab-4502-9780-bbf282f416d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:53.016789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016764 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-credential-socket" pod "78b65921-33ab-4502-9780-bbf282f416d4" (UID: "78b65921-33ab-4502-9780-bbf282f416d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:53.016898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016880 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:53.016965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016901 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:53.016965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016911 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:53.016965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016919 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b65921-33ab-4502-9780-bbf282f416d4-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:53.016965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.016927 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:53.018898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.018874 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-envoy" pod "78b65921-33ab-4502-9780-bbf282f416d4" (UID: "78b65921-33ab-4502-9780-bbf282f416d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:53.018898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.018874 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78b65921-33ab-4502-9780-bbf282f416d4-istio-podinfo" pod "78b65921-33ab-4502-9780-bbf282f416d4" (UID: "78b65921-33ab-4502-9780-bbf282f416d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:06:53.019066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.018907 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-istio-token" pod "78b65921-33ab-4502-9780-bbf282f416d4" (UID: "78b65921-33ab-4502-9780-bbf282f416d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:53.019066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.018972 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-kube-api-access-rxk8m" pod "78b65921-33ab-4502-9780-bbf282f416d4" (UID: "78b65921-33ab-4502-9780-bbf282f416d4"). InnerVolumeSpecName "kube-api-access-rxk8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:53.118187 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.118124 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b65921-33ab-4502-9780-bbf282f416d4-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:53.118187 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.118152 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:53.118187 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.118167 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b65921-33ab-4502-9780-bbf282f416d4-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:53.118187 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.118177 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxk8m\" (UniqueName: \"kubernetes.io/projected/78b65921-33ab-4502-9780-bbf282f416d4-kube-api-access-rxk8m\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:53.294865 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.294835 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj"] Oct 30 06:06:53.295064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.295043 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" containerID="cri-o://87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab" gracePeriod=30 Oct 30 06:06:53.336025 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.335999 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52607b4-c7bf-443a-a2f2-4989d4528e49" path="/var/lib/kubelet/pods/e52607b4-c7bf-443a-a2f2-4989d4528e49/volumes" Oct 30 06:06:53.425560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.425472 4431 patch_prober.go:29] interesting pod/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 06:06:53.425560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.425541 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:06:53.575990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.575957 4431 generic.go:356] "Generic (PLEG): container finished" podID="78b65921-33ab-4502-9780-bbf282f416d4" containerID="4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b" exitCode=0 Oct 30 06:06:53.576140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.576024 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" Oct 30 06:06:53.576140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.576042 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" event={"ID":"78b65921-33ab-4502-9780-bbf282f416d4","Type":"ContainerDied","Data":"4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b"} Oct 30 06:06:53.576257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.576137 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz" event={"ID":"78b65921-33ab-4502-9780-bbf282f416d4","Type":"ContainerDied","Data":"eeca493e3da802f0f3a21ba731e185e5e91514423a4dc0868adb4eca0e7ef342"} Oct 30 06:06:53.576257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.576172 4431 scope.go:122] "RemoveContainer" containerID="4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b" Oct 30 06:06:53.605749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.605730 4431 scope.go:122] "RemoveContainer" containerID="872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434" Oct 30 06:06:53.613814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.613794 4431 scope.go:122] "RemoveContainer" containerID="4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b" Oct 30 06:06:53.614085 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:53.614065 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b\": container with ID starting with 4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b not found: ID does not exist" containerID="4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b" Oct 30 06:06:53.614163 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.614098 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b"} err="failed to get container status \"4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b\": rpc error: code = NotFound desc = could not find container \"4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b\": container with ID starting with 4b30a2d8376e4f5570682b989611d46d1703163c8cc8470f0ddbd3774f6c0d1b not found: ID does not exist" Oct 30 06:06:53.614163 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.614122 4431 scope.go:122] "RemoveContainer" containerID="872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434" Oct 30 06:06:53.614376 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:53.614361 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434\": container with ID starting with 872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434 not found: ID does not exist" containerID="872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434" Oct 30 06:06:53.614414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.614387 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434"} err="failed to get container status \"872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434\": rpc error: code = NotFound desc = could not find container \"872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434\": container with ID starting with 872b768f3b8ac6c9b9b1fadce4b798483cfa67f6e4bf8a9124d8628c38528434 not found: ID does not exist" Oct 30 06:06:53.641228 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.641208 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz"] Oct 30 06:06:53.649354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:53.649333 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c45a2ff8-openshift-default-646b99d9d7-sf9xz"] Oct 30 06:06:54.610840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.610811 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w"] Oct 30 06:06:54.611261 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611246 4431 cpu_manager.go:414] "containerMap: removing container" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" Oct 30 06:06:54.611328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611263 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" Oct 30 06:06:54.611328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611274 4431 cpu_manager.go:414] "containerMap: removing container" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" Oct 30 06:06:54.611328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611279 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" Oct 30 06:06:54.611328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611294 4431 cpu_manager.go:414] "containerMap: removing container" podUID="bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" containerName="maas-api" Oct 30 06:06:54.611328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611313 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611331 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b947f444-7f82-4ccb-b81e-691633617e78" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611336 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b947f444-7f82-4ccb-b81e-691633617e78" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611342 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e52607b4-c7bf-443a-a2f2-4989d4528e49" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611347 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52607b4-c7bf-443a-a2f2-4989d4528e49" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611354 4431 cpu_manager.go:414] "containerMap: removing container" podUID="9c6c4792-8255-4bcb-abb7-6c497ca4a661" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611359 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6c4792-8255-4bcb-abb7-6c497ca4a661" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611430 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6c4792-8255-4bcb-abb7-6c497ca4a661" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611441 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611446 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b65921-33ab-4502-9780-bbf282f416d4" containerName="istio-proxy" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611452 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfbc2e64-14e6-4b45-8ee7-05c1ff65c3f5" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611459 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b947f444-7f82-4ccb-b81e-691633617e78" containerName="maas-api" Oct 30 06:06:54.611496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.611463 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52607b4-c7bf-443a-a2f2-4989d4528e49" containerName="maas-api" Oct 30 06:06:54.614089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.614074 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.643086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.643054 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w"] Oct 30 06:06:54.659425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.659391 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a519e9-openshift-default-dockercfg-nffns\"" Oct 30 06:06:54.729981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.729947 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsvkh\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-kube-api-access-gsvkh\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.730143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.730001 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.730143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.730065 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.730143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.730125 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.730264 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.730166 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.730264 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.730203 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.730264 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.730228 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.730398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.730349 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.730398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.730377 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.789776 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.789745 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt"] Oct 30 06:06:54.792367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.792350 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:06:54.812733 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.812709 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt"] Oct 30 06:06:54.831173 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831141 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831193 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831247 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm44c\" (UniqueName: \"kubernetes.io/projected/3c5462b6-b727-4176-af24-18f12c02d3c2-kube-api-access-cm44c\") pod \"payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt\" (UID: \"3c5462b6-b727-4176-af24-18f12c02d3c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:06:54.831428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831339 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831393 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831424 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831464 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831491 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831530 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831571 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsvkh\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-kube-api-access-gsvkh\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831606 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5462b6-b727-4176-af24-18f12c02d3c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt\" (UID: \"3c5462b6-b727-4176-af24-18f12c02d3c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:06:54.831818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831656 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.831818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.831789 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.832099 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.832081 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.832270 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.832248 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.832716 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.832693 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.833847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.833820 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.834748 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.834726 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.838772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.838747 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8d6349\"" Oct 30 06:06:54.838863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.838776 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8d6349-dockercfg-5cg6v\"" Oct 30 06:06:54.854529 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.854510 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsvkh\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-kube-api-access-gsvkh\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.854630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.854529 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.929056 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.928965 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:54.932789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.932761 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5462b6-b727-4176-af24-18f12c02d3c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt\" (UID: \"3c5462b6-b727-4176-af24-18f12c02d3c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:06:54.932905 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.932849 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm44c\" (UniqueName: \"kubernetes.io/projected/3c5462b6-b727-4176-af24-18f12c02d3c2-kube-api-access-cm44c\") pod \"payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt\" (UID: \"3c5462b6-b727-4176-af24-18f12c02d3c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:06:54.933405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.933388 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5462b6-b727-4176-af24-18f12c02d3c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt\" (UID: \"3c5462b6-b727-4176-af24-18f12c02d3c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:06:54.959729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:54.959703 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm44c\" (UniqueName: \"kubernetes.io/projected/3c5462b6-b727-4176-af24-18f12c02d3c2-kube-api-access-cm44c\") pod \"payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt\" (UID: \"3c5462b6-b727-4176-af24-18f12c02d3c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:06:55.104834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.104801 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:06:55.192441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.192421 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:55.236711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.236685 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-kube-api-access-xq76f\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-kube-api-access-xq76f\") pod \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " Oct 30 06:06:55.236882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.236722 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-certs\") pod \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " Oct 30 06:06:55.236882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.236760 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-envoy\") pod \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " Oct 30 06:06:55.236882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.236781 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-credential-socket\") pod \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " Oct 30 06:06:55.236882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.236805 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-socket\") pod \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " Oct 30 06:06:55.236882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.236872 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istiod-ca-cert\") pod \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " Oct 30 06:06:55.237146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.236900 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-data\") pod \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " Oct 30 06:06:55.237146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.236939 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-token\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-token\") pod \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " Oct 30 06:06:55.237146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.236978 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-podinfo\") pod \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\" (UID: \"fa3444fc-dd5a-401f-892c-5d2a4e5a7746\") " Oct 30 06:06:55.237146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.237065 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-certs" pod "fa3444fc-dd5a-401f-892c-5d2a4e5a7746" (UID: "fa3444fc-dd5a-401f-892c-5d2a4e5a7746"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:55.237146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.237080 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-credential-socket" pod "fa3444fc-dd5a-401f-892c-5d2a4e5a7746" (UID: "fa3444fc-dd5a-401f-892c-5d2a4e5a7746"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:55.237476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.237216 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-socket" pod "fa3444fc-dd5a-401f-892c-5d2a4e5a7746" (UID: "fa3444fc-dd5a-401f-892c-5d2a4e5a7746"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:55.237476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.237387 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:55.237476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.237402 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-data" pod "fa3444fc-dd5a-401f-892c-5d2a4e5a7746" (UID: "fa3444fc-dd5a-401f-892c-5d2a4e5a7746"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:55.237476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.237409 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:55.237476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.237456 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:55.237730 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.237616 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istiod-ca-cert" pod "fa3444fc-dd5a-401f-892c-5d2a4e5a7746" (UID: "fa3444fc-dd5a-401f-892c-5d2a4e5a7746"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:55.239633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.239611 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-kube-api-access-xq76f" pod "fa3444fc-dd5a-401f-892c-5d2a4e5a7746" (UID: "fa3444fc-dd5a-401f-892c-5d2a4e5a7746"). InnerVolumeSpecName "kube-api-access-xq76f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:55.240070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.240010 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-envoy" pod "fa3444fc-dd5a-401f-892c-5d2a4e5a7746" (UID: "fa3444fc-dd5a-401f-892c-5d2a4e5a7746"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:55.240472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.240445 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-token" pod "fa3444fc-dd5a-401f-892c-5d2a4e5a7746" (UID: "fa3444fc-dd5a-401f-892c-5d2a4e5a7746"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:55.240575 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.240445 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-podinfo" pod "fa3444fc-dd5a-401f-892c-5d2a4e5a7746" (UID: "fa3444fc-dd5a-401f-892c-5d2a4e5a7746"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:06:55.251579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.251545 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w"] Oct 30 06:06:55.256737 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:55.256716 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84beb3ad_bff6_4552_b721_dc7a05b4cc92.slice/crio-7f33bf29b9df19f64c291d7087470d0a44cdb91fffa576537bc4a3e2f0bfb817 WatchSource:0}: Error finding container 7f33bf29b9df19f64c291d7087470d0a44cdb91fffa576537bc4a3e2f0bfb817: Status 404 returned error can't find the container with id 7f33bf29b9df19f64c291d7087470d0a44cdb91fffa576537bc4a3e2f0bfb817 Oct 30 06:06:55.259615 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.259584 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:55.259724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.259660 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:55.259724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.259705 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:06:55.336398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.336364 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b65921-33ab-4502-9780-bbf282f416d4" path="/var/lib/kubelet/pods/78b65921-33ab-4502-9780-bbf282f416d4/volumes" Oct 30 06:06:55.338295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.338096 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:55.338295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.338116 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:55.338295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.338128 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:55.338295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.338141 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:55.338295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.338154 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq76f\" (UniqueName: \"kubernetes.io/projected/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-kube-api-access-xq76f\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:55.338295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.338168 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3444fc-dd5a-401f-892c-5d2a4e5a7746-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:55.416631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.416607 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt"] Oct 30 06:06:55.418563 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:55.418526 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5462b6_b727_4176_af24_18f12c02d3c2.slice/crio-a27a9b9b485e94b64fef07c3a737712e36de584e032b2afd7e90724183c93603 WatchSource:0}: Error finding container a27a9b9b485e94b64fef07c3a737712e36de584e032b2afd7e90724183c93603: Status 404 returned error can't find the container with id a27a9b9b485e94b64fef07c3a737712e36de584e032b2afd7e90724183c93603 Oct 30 06:06:55.589245 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.589202 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" event={"ID":"3c5462b6-b727-4176-af24-18f12c02d3c2","Type":"ContainerStarted","Data":"3d731db4a89fa9b9efe99eb7f730a10faec5d68ad63f810a680c255e0038f97d"} Oct 30 06:06:55.589427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.589265 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" event={"ID":"3c5462b6-b727-4176-af24-18f12c02d3c2","Type":"ContainerStarted","Data":"a27a9b9b485e94b64fef07c3a737712e36de584e032b2afd7e90724183c93603"} Oct 30 06:06:55.589427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.589363 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:06:55.590854 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.590834 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" event={"ID":"84beb3ad-bff6-4552-b721-dc7a05b4cc92","Type":"ContainerStarted","Data":"a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48"} Oct 30 06:06:55.590973 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.590867 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" event={"ID":"84beb3ad-bff6-4552-b721-dc7a05b4cc92","Type":"ContainerStarted","Data":"7f33bf29b9df19f64c291d7087470d0a44cdb91fffa576537bc4a3e2f0bfb817"} Oct 30 06:06:55.592387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.592366 4431 generic.go:356] "Generic (PLEG): container finished" podID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerID="27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e" exitCode=0 Oct 30 06:06:55.592472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.592403 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" event={"ID":"fa3444fc-dd5a-401f-892c-5d2a4e5a7746","Type":"ContainerDied","Data":"27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e"} Oct 30 06:06:55.592472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.592409 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" Oct 30 06:06:55.592472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.592435 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf" event={"ID":"fa3444fc-dd5a-401f-892c-5d2a4e5a7746","Type":"ContainerDied","Data":"72601da643083274738ca7b10cfd44c9a45b226660c6dbfcc4e32f57a39fe38f"} Oct 30 06:06:55.592472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.592457 4431 scope.go:122] "RemoveContainer" containerID="27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e" Oct 30 06:06:55.601260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.601238 4431 scope.go:122] "RemoveContainer" containerID="1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90" Oct 30 06:06:55.609111 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.609096 4431 scope.go:122] "RemoveContainer" containerID="27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e" Oct 30 06:06:55.609407 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:55.609387 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e\": container with ID starting with 27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e not found: ID does not exist" containerID="27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e" Oct 30 06:06:55.609462 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.609422 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e"} err="failed to get container status \"27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e\": rpc error: code = NotFound desc = could not find container \"27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e\": container with ID starting with 27438fd182f14852557e6df7a04cefa66c557623b9fcdd91d3ed000ab93e056e not found: ID does not exist" Oct 30 06:06:55.609462 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.609439 4431 scope.go:122] "RemoveContainer" containerID="1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90" Oct 30 06:06:55.609686 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:55.609669 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90\": container with ID starting with 1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90 not found: ID does not exist" containerID="1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90" Oct 30 06:06:55.609725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.609692 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90"} err="failed to get container status \"1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90\": rpc error: code = NotFound desc = could not find container \"1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90\": container with ID starting with 1bc16be1bc77a738bf3c2a03af9d6bb390555518c383cc50391605bd281b3f90 not found: ID does not exist" Oct 30 06:06:55.629584 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.629547 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" podStartSLOduration=1.629535276 podStartE2EDuration="1.629535276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:55.62855388 +0000 UTC m=+824.435075627" watchObservedRunningTime="2026-10-30 06:06:55.629535276 +0000 UTC m=+824.436057023" Oct 30 06:06:55.666053 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.666019 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf"] Oct 30 06:06:55.675792 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.671293 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9c94fbc2-openshift-default-64f47b77d9-cc5bf"] Oct 30 06:06:55.718689 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.718614 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" podStartSLOduration=1.7186022730000001 podStartE2EDuration="1.718602273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:55.717147916 +0000 UTC m=+824.523669664" watchObservedRunningTime="2026-10-30 06:06:55.718602273 +0000 UTC m=+824.525124020" Oct 30 06:06:55.929712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:55.929682 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:56.341614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.341581 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9"] Oct 30 06:06:56.341987 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.341975 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" Oct 30 06:06:56.342030 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.341988 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" Oct 30 06:06:56.342030 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.341999 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" Oct 30 06:06:56.342030 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.342005 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" Oct 30 06:06:56.342126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.342081 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" Oct 30 06:06:56.342126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.342089 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" containerName="istio-proxy" Oct 30 06:06:56.344964 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.344944 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:06:56.368715 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.368688 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-aaf467\"" Oct 30 06:06:56.368848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.368812 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-aaf467-dockercfg-mzv85\"" Oct 30 06:06:56.370779 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.370747 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9"] Oct 30 06:06:56.450149 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.450104 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb21f6c-d934-451c-ae6e-e878af83cc49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9\" (UID: \"0cb21f6c-d934-451c-ae6e-e878af83cc49\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:06:56.450352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.450230 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z267\" (UniqueName: \"kubernetes.io/projected/0cb21f6c-d934-451c-ae6e-e878af83cc49-kube-api-access-6z267\") pod \"payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9\" (UID: \"0cb21f6c-d934-451c-ae6e-e878af83cc49\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:06:56.551091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.551055 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb21f6c-d934-451c-ae6e-e878af83cc49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9\" (UID: \"0cb21f6c-d934-451c-ae6e-e878af83cc49\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:06:56.551091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.551096 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z267\" (UniqueName: \"kubernetes.io/projected/0cb21f6c-d934-451c-ae6e-e878af83cc49-kube-api-access-6z267\") pod \"payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9\" (UID: \"0cb21f6c-d934-451c-ae6e-e878af83cc49\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:06:56.551785 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.551765 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb21f6c-d934-451c-ae6e-e878af83cc49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9\" (UID: \"0cb21f6c-d934-451c-ae6e-e878af83cc49\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:06:56.574155 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.574124 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z267\" (UniqueName: \"kubernetes.io/projected/0cb21f6c-d934-451c-ae6e-e878af83cc49-kube-api-access-6z267\") pod \"payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9\" (UID: \"0cb21f6c-d934-451c-ae6e-e878af83cc49\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:06:56.656479 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.656411 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:06:56.785134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.785097 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs"] Oct 30 06:06:56.788191 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.788168 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.802442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.802420 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-8d6349-serving-cert\"" Oct 30 06:06:56.809891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.809867 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs"] Oct 30 06:06:56.854884 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.854846 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-metrics-certs\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.855051 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.854899 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0402-ae16-458d-bdae-659aa77365d9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.855051 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.855039 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzrc\" (UniqueName: \"kubernetes.io/projected/b81c0402-ae16-458d-bdae-659aa77365d9-kube-api-access-lpzrc\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.855191 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.855161 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.934632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.934571 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:56.955733 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.955713 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9"] Oct 30 06:06:56.956887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.956090 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0402-ae16-458d-bdae-659aa77365d9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.956887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.956172 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzrc\" (UniqueName: \"kubernetes.io/projected/b81c0402-ae16-458d-bdae-659aa77365d9-kube-api-access-lpzrc\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.956887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.956315 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.956887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.956384 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-metrics-certs\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.956887 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:56.956408 4431 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-serving-cert: secret "maas-api-e2e-worker-w3-8d6349-serving-cert" not found Oct 30 06:06:56.956887 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:56.956474 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls podName:b81c0402-ae16-458d-bdae-659aa77365d9 nodeName:}" failed. No retries permitted until 2026-10-30 06:06:57.456454517 +0000 UTC m=+826.262976242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls") pod "maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" (UID: "b81c0402-ae16-458d-bdae-659aa77365d9") : secret "maas-api-e2e-worker-w3-8d6349-serving-cert" not found Oct 30 06:06:56.956887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.956848 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0402-ae16-458d-bdae-659aa77365d9-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.958767 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:56.958730 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb21f6c_d934_451c_ae6e_e878af83cc49.slice/crio-c6ce3942aac27494d2427c39b53b3c52ac02fa76561586b9804d0fe212c2b53f WatchSource:0}: Error finding container c6ce3942aac27494d2427c39b53b3c52ac02fa76561586b9804d0fe212c2b53f: Status 404 returned error can't find the container with id c6ce3942aac27494d2427c39b53b3c52ac02fa76561586b9804d0fe212c2b53f Oct 30 06:06:56.958939 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.958922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-metrics-certs\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:56.981132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:56.981093 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzrc\" (UniqueName: \"kubernetes.io/projected/b81c0402-ae16-458d-bdae-659aa77365d9-kube-api-access-lpzrc\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:57.343550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.343510 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3444fc-dd5a-401f-892c-5d2a4e5a7746" path="/var/lib/kubelet/pods/fa3444fc-dd5a-401f-892c-5d2a4e5a7746/volumes" Oct 30 06:06:57.462916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.462875 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:57.465415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.465387 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:57.603786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.603709 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" event={"ID":"0cb21f6c-d934-451c-ae6e-e878af83cc49","Type":"ContainerStarted","Data":"1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9"} Oct 30 06:06:57.603786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.603764 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" event={"ID":"0cb21f6c-d934-451c-ae6e-e878af83cc49","Type":"ContainerStarted","Data":"c6ce3942aac27494d2427c39b53b3c52ac02fa76561586b9804d0fe212c2b53f"} Oct 30 06:06:57.603956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.603896 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:57.604901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.604882 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:06:57.645531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.645476 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" podStartSLOduration=1.645463023 podStartE2EDuration="1.645463023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:57.643431978 +0000 UTC m=+826.449953726" watchObservedRunningTime="2026-10-30 06:06:57.645463023 +0000 UTC m=+826.451984771" Oct 30 06:06:57.700722 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.700692 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:57.949696 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.949669 4431 patch_prober.go:29] interesting pod/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 06:06:57.949861 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:57.949720 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:06:58.008378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.008351 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs"] Oct 30 06:06:58.010679 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:58.010650 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81c0402_ae16_458d_bdae_659aa77365d9.slice/crio-cfe60325ce9348c2e13dfe727a7a6b440fad1fd04fc17a9d168fac354f7672dc WatchSource:0}: Error finding container cfe60325ce9348c2e13dfe727a7a6b440fad1fd04fc17a9d168fac354f7672dc: Status 404 returned error can't find the container with id cfe60325ce9348c2e13dfe727a7a6b440fad1fd04fc17a9d168fac354f7672dc Oct 30 06:06:58.548435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.548411 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:58.609757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.609687 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" event={"ID":"b81c0402-ae16-458d-bdae-659aa77365d9","Type":"ContainerStarted","Data":"78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416"} Oct 30 06:06:58.609757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.609743 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" event={"ID":"b81c0402-ae16-458d-bdae-659aa77365d9","Type":"ContainerStarted","Data":"cfe60325ce9348c2e13dfe727a7a6b440fad1fd04fc17a9d168fac354f7672dc"} Oct 30 06:06:58.609918 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.609768 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:06:58.611313 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.611275 4431 generic.go:356] "Generic (PLEG): container finished" podID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerID="87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab" exitCode=0 Oct 30 06:06:58.611410 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.611338 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" Oct 30 06:06:58.611410 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.611341 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" event={"ID":"e2afa81d-0896-4648-b9d3-0ee402571a09","Type":"ContainerDied","Data":"87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab"} Oct 30 06:06:58.611410 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.611392 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj" event={"ID":"e2afa81d-0896-4648-b9d3-0ee402571a09","Type":"ContainerDied","Data":"209ea95c3c7fb33a85b6e042db059f4ad37407e4fa721c91d1a5222c8fb9c5ee"} Oct 30 06:06:58.611518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.611419 4431 scope.go:122] "RemoveContainer" containerID="87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab" Oct 30 06:06:58.611769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.611752 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:06:58.623105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.623077 4431 scope.go:122] "RemoveContainer" containerID="f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e" Oct 30 06:06:58.634376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.634355 4431 scope.go:122] "RemoveContainer" containerID="87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab" Oct 30 06:06:58.634695 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:58.634666 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab\": container with ID starting with 87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab not found: ID does not exist" containerID="87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab" Oct 30 06:06:58.634784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.634692 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab"} err="failed to get container status \"87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab\": rpc error: code = NotFound desc = could not find container \"87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab\": container with ID starting with 87149a596d6c25a894b60f5f99948a1cd93da59449a76040b6a025d166103cab not found: ID does not exist" Oct 30 06:06:58.634784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.634712 4431 scope.go:122] "RemoveContainer" containerID="f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e" Oct 30 06:06:58.634949 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:06:58.634926 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e\": container with ID starting with f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e not found: ID does not exist" containerID="f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e" Oct 30 06:06:58.634991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.634948 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e"} err="failed to get container status \"f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e\": rpc error: code = NotFound desc = could not find container \"f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e\": container with ID starting with f4f3b8c20aa001b8441f8220658ffe567d56edd79ea0e14e32f90b04c94da53e not found: ID does not exist" Oct 30 06:06:58.657564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.657523 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" podStartSLOduration=2.3239368479999998 podStartE2EDuration="2.657511309s" totalImagesPullingTime="333.574461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:58.656036943 +0000 UTC m=+827.462558693" watchObservedRunningTime="2026-10-30 06:06:58.657511309 +0000 UTC m=+827.464033057" Oct 30 06:06:58.678662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.678637 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-envoy\") pod \"e2afa81d-0896-4648-b9d3-0ee402571a09\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " Oct 30 06:06:58.678756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.678671 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-socket\") pod \"e2afa81d-0896-4648-b9d3-0ee402571a09\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " Oct 30 06:06:58.678756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.678727 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-token\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-token\") pod \"e2afa81d-0896-4648-b9d3-0ee402571a09\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " Oct 30 06:06:58.678828 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.678776 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2afa81d-0896-4648-b9d3-0ee402571a09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2afa81d-0896-4648-b9d3-0ee402571a09-istiod-ca-cert\") pod \"e2afa81d-0896-4648-b9d3-0ee402571a09\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " Oct 30 06:06:58.678828 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.678806 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-certs\") pod \"e2afa81d-0896-4648-b9d3-0ee402571a09\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " Oct 30 06:06:58.678922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.678884 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-credential-socket\") pod \"e2afa81d-0896-4648-b9d3-0ee402571a09\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " Oct 30 06:06:58.678922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.678912 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-podinfo\") pod \"e2afa81d-0896-4648-b9d3-0ee402571a09\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " Oct 30 06:06:58.679020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.678944 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-data\") pod \"e2afa81d-0896-4648-b9d3-0ee402571a09\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " Oct 30 06:06:58.679020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.679004 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-socket" pod "e2afa81d-0896-4648-b9d3-0ee402571a09" (UID: "e2afa81d-0896-4648-b9d3-0ee402571a09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:58.679128 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.679078 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-kube-api-access-hf5tv\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-kube-api-access-hf5tv\") pod \"e2afa81d-0896-4648-b9d3-0ee402571a09\" (UID: \"e2afa81d-0896-4648-b9d3-0ee402571a09\") " Oct 30 06:06:58.679724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.679175 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-credential-socket" pod "e2afa81d-0896-4648-b9d3-0ee402571a09" (UID: "e2afa81d-0896-4648-b9d3-0ee402571a09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:58.679724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.679183 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-certs" pod "e2afa81d-0896-4648-b9d3-0ee402571a09" (UID: "e2afa81d-0896-4648-b9d3-0ee402571a09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:58.679724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.679271 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2afa81d-0896-4648-b9d3-0ee402571a09-istiod-ca-cert" pod "e2afa81d-0896-4648-b9d3-0ee402571a09" (UID: "e2afa81d-0896-4648-b9d3-0ee402571a09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:58.679724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.679434 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-data" pod "e2afa81d-0896-4648-b9d3-0ee402571a09" (UID: "e2afa81d-0896-4648-b9d3-0ee402571a09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:58.680013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.679885 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:58.680230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.680189 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2afa81d-0896-4648-b9d3-0ee402571a09-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:58.680230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.680218 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:58.680432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.680237 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:58.680432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.680252 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:58.681354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.681321 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-envoy" pod "e2afa81d-0896-4648-b9d3-0ee402571a09" (UID: "e2afa81d-0896-4648-b9d3-0ee402571a09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:58.681446 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.681397 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-token" pod "e2afa81d-0896-4648-b9d3-0ee402571a09" (UID: "e2afa81d-0896-4648-b9d3-0ee402571a09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:58.681446 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.681408 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-kube-api-access-hf5tv" pod "e2afa81d-0896-4648-b9d3-0ee402571a09" (UID: "e2afa81d-0896-4648-b9d3-0ee402571a09"). InnerVolumeSpecName "kube-api-access-hf5tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:58.681791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.681772 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-podinfo" pod "e2afa81d-0896-4648-b9d3-0ee402571a09" (UID: "e2afa81d-0896-4648-b9d3-0ee402571a09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:06:58.781640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.781595 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf5tv\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-kube-api-access-hf5tv\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:58.781640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.781631 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:58.781640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.781646 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:58.782453 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.781659 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2afa81d-0896-4648-b9d3-0ee402571a09-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:06:58.889762 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.889688 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt"] Oct 30 06:06:58.890107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.890096 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" Oct 30 06:06:58.890145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.890109 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" Oct 30 06:06:58.890145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.890121 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" Oct 30 06:06:58.890145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.890126 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" Oct 30 06:06:58.890251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.890184 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" Oct 30 06:06:58.894929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.894906 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:58.913418 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.913393 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-aaf467-serving-cert\"" Oct 30 06:06:58.928152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.928131 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt"] Oct 30 06:06:58.983748 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.983716 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-metrics-certs\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:58.983869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.983791 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d58406-ef62-4c3e-a222-91f79c761619-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:58.983869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.983843 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt527\" (UniqueName: \"kubernetes.io/projected/13d58406-ef62-4c3e-a222-91f79c761619-kube-api-access-jt527\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:58.983947 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.983871 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-maas-api-tls\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:58.998572 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:58.998545 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj"] Oct 30 06:06:59.007618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.007593 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-55140dc4-openshift-default-cd74ccd77-2tvbj"] Oct 30 06:06:59.085161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.085134 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt527\" (UniqueName: \"kubernetes.io/projected/13d58406-ef62-4c3e-a222-91f79c761619-kube-api-access-jt527\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:59.085275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.085169 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-maas-api-tls\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:59.085275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.085235 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-metrics-certs\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:59.085427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.085290 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d58406-ef62-4c3e-a222-91f79c761619-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:59.085987 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.085964 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d58406-ef62-4c3e-a222-91f79c761619-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:59.087701 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.087681 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-metrics-certs\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:59.087789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.087727 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-maas-api-tls\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:59.110512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.110492 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt527\" (UniqueName: \"kubernetes.io/projected/13d58406-ef62-4c3e-a222-91f79c761619-kube-api-access-jt527\") pod \"maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:59.207879 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.207847 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:06:59.342686 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.342655 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" path="/var/lib/kubelet/pods/e2afa81d-0896-4648-b9d3-0ee402571a09/volumes" Oct 30 06:06:59.522415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.522339 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt"] Oct 30 06:06:59.524642 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:06:59.524617 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d58406_ef62_4c3e_a222_91f79c761619.slice/crio-dc3910cc9c0d00c50bc5dea089a021510110860e26f0c37dc70d02c397c20537 WatchSource:0}: Error finding container dc3910cc9c0d00c50bc5dea089a021510110860e26f0c37dc70d02c397c20537: Status 404 returned error can't find the container with id dc3910cc9c0d00c50bc5dea089a021510110860e26f0c37dc70d02c397c20537 Oct 30 06:06:59.619564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:06:59.619523 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" event={"ID":"13d58406-ef62-4c3e-a222-91f79c761619","Type":"ContainerStarted","Data":"dc3910cc9c0d00c50bc5dea089a021510110860e26f0c37dc70d02c397c20537"} Oct 30 06:07:00.627727 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:00.627690 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" event={"ID":"13d58406-ef62-4c3e-a222-91f79c761619","Type":"ContainerStarted","Data":"ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf"} Oct 30 06:07:00.628212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:00.627837 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:07:02.061723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.061663 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" podStartSLOduration=3.6138035779999997 podStartE2EDuration="4.061648026s" totalImagesPullingTime="447.844448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:00.669731316 +0000 UTC m=+829.476253066" watchObservedRunningTime="2026-10-30 06:07:02.061648026 +0000 UTC m=+830.868169775" Oct 30 06:07:02.062433 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.062049 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7"] Oct 30 06:07:02.062804 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.062781 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2afa81d-0896-4648-b9d3-0ee402571a09" containerName="istio-proxy" Oct 30 06:07:02.066170 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.066148 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:02.096675 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.096651 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7"] Oct 30 06:07:02.112219 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.112191 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7xd\" (UniqueName: \"kubernetes.io/projected/96a04f24-d9ec-472f-8ec2-002b5cabf3da-kube-api-access-xz7xd\") pod \"payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7\" (UID: \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:02.112385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.112250 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a04f24-d9ec-472f-8ec2-002b5cabf3da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7\" (UID: \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:02.143121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.142515 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a519e9-dockercfg-mq22m\"" Oct 30 06:07:02.154140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.154107 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a519e9\"" Oct 30 06:07:02.213083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.213047 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7xd\" (UniqueName: \"kubernetes.io/projected/96a04f24-d9ec-472f-8ec2-002b5cabf3da-kube-api-access-xz7xd\") pod \"payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7\" (UID: \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:02.213279 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.213095 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a04f24-d9ec-472f-8ec2-002b5cabf3da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7\" (UID: \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:02.213916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.213893 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a04f24-d9ec-472f-8ec2-002b5cabf3da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7\" (UID: \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:02.236568 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.236539 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7xd\" (UniqueName: \"kubernetes.io/projected/96a04f24-d9ec-472f-8ec2-002b5cabf3da-kube-api-access-xz7xd\") pod \"payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7\" (UID: \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:02.383859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.383780 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:02.683295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:02.683271 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7"] Oct 30 06:07:02.685486 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:07:02.685451 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a04f24_d9ec_472f_8ec2_002b5cabf3da.slice/crio-94603a0b7b3e7178b2107708f6259ba7c6ba60206ea1f85f1c7cf844eee51b8b WatchSource:0}: Error finding container 94603a0b7b3e7178b2107708f6259ba7c6ba60206ea1f85f1c7cf844eee51b8b: Status 404 returned error can't find the container with id 94603a0b7b3e7178b2107708f6259ba7c6ba60206ea1f85f1c7cf844eee51b8b Oct 30 06:07:03.061003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.060969 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w"] Oct 30 06:07:03.064692 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.064669 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.079448 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.079424 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-eaee88-serving-cert\"" Oct 30 06:07:03.087162 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.087134 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w"] Oct 30 06:07:03.124140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.124109 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-metrics-certs\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.124278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.124234 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d96bcc0-783d-466d-9f38-820842153c58-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.124278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.124264 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69bz\" (UniqueName: \"kubernetes.io/projected/9d96bcc0-783d-466d-9f38-820842153c58-kube-api-access-j69bz\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.124405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.124295 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.225734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.225705 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d96bcc0-783d-466d-9f38-820842153c58-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.225734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.225736 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69bz\" (UniqueName: \"kubernetes.io/projected/9d96bcc0-783d-466d-9f38-820842153c58-kube-api-access-j69bz\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.225927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.225763 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.225927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.225833 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-metrics-certs\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.226361 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.226329 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d96bcc0-783d-466d-9f38-820842153c58-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.228277 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.228252 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.228428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.228408 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-metrics-certs\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.248863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.248838 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69bz\" (UniqueName: \"kubernetes.io/projected/9d96bcc0-783d-466d-9f38-820842153c58-kube-api-access-j69bz\") pod \"maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.381979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.381903 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:03.643430 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.643337 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" event={"ID":"96a04f24-d9ec-472f-8ec2-002b5cabf3da","Type":"ContainerStarted","Data":"5bb8a6e73658cb32b9c71b66b3ffd28669e32c0bca0a9886abad3e96d6e16321"} Oct 30 06:07:03.643430 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.643393 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" event={"ID":"96a04f24-d9ec-472f-8ec2-002b5cabf3da","Type":"ContainerStarted","Data":"94603a0b7b3e7178b2107708f6259ba7c6ba60206ea1f85f1c7cf844eee51b8b"} Oct 30 06:07:03.643671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.643452 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:03.683593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.683549 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w"] Oct 30 06:07:03.685175 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:07:03.685142 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d96bcc0_783d_466d_9f38_820842153c58.slice/crio-d4bc1c0e001947fec1ed0ffdb46689ce1814b994c6edee90330d5253aae7cf9c WatchSource:0}: Error finding container d4bc1c0e001947fec1ed0ffdb46689ce1814b994c6edee90330d5253aae7cf9c: Status 404 returned error can't find the container with id d4bc1c0e001947fec1ed0ffdb46689ce1814b994c6edee90330d5253aae7cf9c Oct 30 06:07:03.693907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:03.693860 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" podStartSLOduration=1.693843523 podStartE2EDuration="1.693843523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:03.683060522 +0000 UTC m=+832.489582270" watchObservedRunningTime="2026-10-30 06:07:03.693843523 +0000 UTC m=+832.500365271" Oct 30 06:07:03.702460 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:07:03.702434 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:07:04.624706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:04.624678 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 06:07:04.650217 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:04.650184 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" event={"ID":"9d96bcc0-783d-466d-9f38-820842153c58","Type":"ContainerStarted","Data":"799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04"} Oct 30 06:07:04.650380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:04.650236 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" event={"ID":"9d96bcc0-783d-466d-9f38-820842153c58","Type":"ContainerStarted","Data":"d4bc1c0e001947fec1ed0ffdb46689ce1814b994c6edee90330d5253aae7cf9c"} Oct 30 06:07:04.650439 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:04.650377 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:04.708806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:04.708750 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" podStartSLOduration=1.406897182 podStartE2EDuration="1.708733395s" totalImagesPullingTime="301.836213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:04.708351891 +0000 UTC m=+833.514873642" watchObservedRunningTime="2026-10-30 06:07:04.708733395 +0000 UTC m=+833.515255143" Oct 30 06:07:04.737801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:04.737770 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl"] Oct 30 06:07:04.738046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:04.738006 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" podUID="8ab5cc92-9099-48d0-913b-c8b4ff7160ff" containerName="maas-api" containerID="cri-o://fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027" gracePeriod=30 Oct 30 06:07:04.979574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:04.979545 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:07:05.043228 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.043193 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-metrics-certs\") pod \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " Oct 30 06:07:05.043435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.043338 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-kube-api-access-qmlgd\" (UniqueName: \"kubernetes.io/projected/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-kube-api-access-qmlgd\") pod \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " Oct 30 06:07:05.043435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.043408 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls\") pod \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " Oct 30 06:07:05.043631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.043445 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-openshift-service-ca\") pod \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\" (UID: \"8ab5cc92-9099-48d0-913b-c8b4ff7160ff\") " Oct 30 06:07:05.044026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.043994 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-openshift-service-ca" pod "8ab5cc92-9099-48d0-913b-c8b4ff7160ff" (UID: "8ab5cc92-9099-48d0-913b-c8b4ff7160ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:05.045534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.045498 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-kube-api-access-qmlgd" pod "8ab5cc92-9099-48d0-913b-c8b4ff7160ff" (UID: "8ab5cc92-9099-48d0-913b-c8b4ff7160ff"). InnerVolumeSpecName "kube-api-access-qmlgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:05.045672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.045648 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-metrics-certs" pod "8ab5cc92-9099-48d0-913b-c8b4ff7160ff" (UID: "8ab5cc92-9099-48d0-913b-c8b4ff7160ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:05.045789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.045750 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls" pod "8ab5cc92-9099-48d0-913b-c8b4ff7160ff" (UID: "8ab5cc92-9099-48d0-913b-c8b4ff7160ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:05.144792 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.144767 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:07:05.144792 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.144790 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:07:05.144941 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.144800 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:07:05.144941 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.144808 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlgd\" (UniqueName: \"kubernetes.io/projected/8ab5cc92-9099-48d0-913b-c8b4ff7160ff-kube-api-access-qmlgd\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:07:05.656243 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.656159 4431 generic.go:356] "Generic (PLEG): container finished" podID="8ab5cc92-9099-48d0-913b-c8b4ff7160ff" containerID="fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027" exitCode=0 Oct 30 06:07:05.656243 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.656217 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" Oct 30 06:07:05.656816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.656244 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" event={"ID":"8ab5cc92-9099-48d0-913b-c8b4ff7160ff","Type":"ContainerDied","Data":"fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027"} Oct 30 06:07:05.656816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.656313 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl" event={"ID":"8ab5cc92-9099-48d0-913b-c8b4ff7160ff","Type":"ContainerDied","Data":"d4a8d08d1103d52191377322e44f919c29a361d9db644aa6b97b400681ab7e71"} Oct 30 06:07:05.656816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.656330 4431 scope.go:122] "RemoveContainer" containerID="fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027" Oct 30 06:07:05.664911 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.664895 4431 scope.go:122] "RemoveContainer" containerID="fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027" Oct 30 06:07:05.665191 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:07:05.665165 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027\": container with ID starting with fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027 not found: ID does not exist" containerID="fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027" Oct 30 06:07:05.665259 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.665194 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027"} err="failed to get container status \"fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027\": rpc error: code = NotFound desc = could not find container \"fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027\": container with ID starting with fd59a055d2fc95ae6981c93f0a55c86b5bd67699994a3b8369b59c16df8ca027 not found: ID does not exist" Oct 30 06:07:05.745752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.745721 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl"] Oct 30 06:07:05.756243 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:05.756218 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3627d700-7945bfc496-fqbjl"] Oct 30 06:07:06.600851 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:06.600823 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 06:07:06.636926 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:06.636901 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 06:07:07.339597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.339569 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab5cc92-9099-48d0-913b-c8b4ff7160ff" path="/var/lib/kubelet/pods/8ab5cc92-9099-48d0-913b-c8b4ff7160ff/volumes" Oct 30 06:07:07.717055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.717023 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k"] Oct 30 06:07:07.717495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.717478 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8ab5cc92-9099-48d0-913b-c8b4ff7160ff" containerName="maas-api" Oct 30 06:07:07.717495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.717495 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab5cc92-9099-48d0-913b-c8b4ff7160ff" containerName="maas-api" Oct 30 06:07:07.717621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.717570 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab5cc92-9099-48d0-913b-c8b4ff7160ff" containerName="maas-api" Oct 30 06:07:07.722107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.722089 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.742736 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.742708 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a519e9-serving-cert\"" Oct 30 06:07:07.758465 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.758434 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k"] Oct 30 06:07:07.874621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.874587 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-metrics-certs\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.874621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.874624 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hn5h\" (UniqueName: \"kubernetes.io/projected/a2c503db-91d1-47a9-a755-72fa60146690-kube-api-access-6hn5h\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.874833 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.874652 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.874878 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.874834 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c503db-91d1-47a9-a755-72fa60146690-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.975730 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.975646 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-metrics-certs\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.975730 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.975694 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hn5h\" (UniqueName: \"kubernetes.io/projected/a2c503db-91d1-47a9-a755-72fa60146690-kube-api-access-6hn5h\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.975980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.975738 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.975980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.975803 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c503db-91d1-47a9-a755-72fa60146690-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.976573 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.976542 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c503db-91d1-47a9-a755-72fa60146690-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.978548 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.978520 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-metrics-certs\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.978783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.978758 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:07.999249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:07.999219 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hn5h\" (UniqueName: \"kubernetes.io/projected/a2c503db-91d1-47a9-a755-72fa60146690-kube-api-access-6hn5h\") pod \"maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:08.034227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:08.034204 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:08.364987 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:08.364962 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k"] Oct 30 06:07:08.366920 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:07:08.366882 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c503db_91d1_47a9_a755_72fa60146690.slice/crio-346604b7ac4a562b7f8b82b527f11b20d5122b7aaf8c5b5f7c37138ef4831d13 WatchSource:0}: Error finding container 346604b7ac4a562b7f8b82b527f11b20d5122b7aaf8c5b5f7c37138ef4831d13: Status 404 returned error can't find the container with id 346604b7ac4a562b7f8b82b527f11b20d5122b7aaf8c5b5f7c37138ef4831d13 Oct 30 06:07:08.675658 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:08.675626 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" event={"ID":"a2c503db-91d1-47a9-a755-72fa60146690","Type":"ContainerStarted","Data":"346604b7ac4a562b7f8b82b527f11b20d5122b7aaf8c5b5f7c37138ef4831d13"} Oct 30 06:07:09.622111 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:09.622081 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 06:07:09.683714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:09.683668 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" event={"ID":"a2c503db-91d1-47a9-a755-72fa60146690","Type":"ContainerStarted","Data":"617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc"} Oct 30 06:07:09.683896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:09.683774 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:09.726805 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:09.726753 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" podStartSLOduration=2.42431786 podStartE2EDuration="2.726740423s" totalImagesPullingTime="302.422563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:09.725273747 +0000 UTC m=+838.531795497" watchObservedRunningTime="2026-10-30 06:07:09.726740423 +0000 UTC m=+838.533262171" Oct 30 06:07:10.662326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:10.662269 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:07:14.654176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:14.654145 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:07:15.035375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.035344 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn"] Oct 30 06:07:15.040561 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.040534 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.057708 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.057665 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"openshift-service-ca.crt\"" Oct 30 06:07:15.057864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.057720 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"kube-root-ca.crt\"" Oct 30 06:07:15.088343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.087762 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn"] Oct 30 06:07:15.130135 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.130103 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"simulator-w3-8d6349-kserve-self-signed-certs\"" Oct 30 06:07:15.130279 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.130149 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"default-dockercfg-xkpcw\"" Oct 30 06:07:15.146575 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.146547 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-dshm\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.146706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.146578 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-model-cache\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.146706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.146614 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-home\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.146706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.146676 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80db6c09-1940-4274-a21c-57ab7dd6738c-tls-certs\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.146851 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.146799 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqmpq\" (UniqueName: \"kubernetes.io/projected/80db6c09-1940-4274-a21c-57ab7dd6738c-kube-api-access-fqmpq\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.146896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.146869 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-tmp-dir\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.248090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.248055 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-dshm\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.248090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.248088 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-model-cache\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.248358 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.248122 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-home\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.248358 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.248151 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80db6c09-1940-4274-a21c-57ab7dd6738c-tls-certs\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.248358 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.248197 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqmpq\" (UniqueName: \"kubernetes.io/projected/80db6c09-1940-4274-a21c-57ab7dd6738c-kube-api-access-fqmpq\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.248358 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.248258 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-tmp-dir\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.248613 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.248591 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-home\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.248677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.248647 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-model-cache\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.248677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.248662 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-tmp-dir\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.250387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.250370 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-dshm\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.250551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.250536 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80db6c09-1940-4274-a21c-57ab7dd6738c-tls-certs\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.287293 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.287200 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqmpq\" (UniqueName: \"kubernetes.io/projected/80db6c09-1940-4274-a21c-57ab7dd6738c-kube-api-access-fqmpq\") pod \"simulator-w3-8d6349-kserve-75b566966b-fpfzn\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.353617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.353587 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:15.686844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.686813 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn"] Oct 30 06:07:15.688270 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:07:15.688242 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80db6c09_1940_4274_a21c_57ab7dd6738c.slice/crio-9260af1c6a23aeae3e38fc40f7fcd71f2aa51b76f9d345f6c7592e31ea0702db WatchSource:0}: Error finding container 9260af1c6a23aeae3e38fc40f7fcd71f2aa51b76f9d345f6c7592e31ea0702db: Status 404 returned error can't find the container with id 9260af1c6a23aeae3e38fc40f7fcd71f2aa51b76f9d345f6c7592e31ea0702db Oct 30 06:07:15.694580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.694562 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:07:15.714634 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:15.714610 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" event={"ID":"80db6c09-1940-4274-a21c-57ab7dd6738c","Type":"ContainerStarted","Data":"9260af1c6a23aeae3e38fc40f7fcd71f2aa51b76f9d345f6c7592e31ea0702db"} Oct 30 06:07:16.720936 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:16.720901 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" event={"ID":"80db6c09-1940-4274-a21c-57ab7dd6738c","Type":"ContainerStarted","Data":"fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978"} Oct 30 06:07:17.354429 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:17.354395 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:20.909767 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:20.909696 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" podStartSLOduration=5.909673383 podStartE2EDuration="5.909673383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:16.783070691 +0000 UTC m=+845.589592441" watchObservedRunningTime="2026-10-30 06:07:20.909673383 +0000 UTC m=+849.716195131" Oct 30 06:07:20.910764 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:20.910740 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp"] Oct 30 06:07:20.915278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:20.915249 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:20.930139 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:20.930113 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-8d44dd-serving-cert\"" Oct 30 06:07:20.946631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:20.946600 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp"] Oct 30 06:07:21.028213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.028174 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-metrics-certs\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.028432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.028234 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54384483-f540-422a-bb19-617300c9b0a8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.028432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.028256 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9m8t\" (UniqueName: \"kubernetes.io/projected/54384483-f540-422a-bb19-617300c9b0a8-kube-api-access-w9m8t\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.028432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.028339 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.130945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.130896 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-metrics-certs\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.131147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.130974 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54384483-f540-422a-bb19-617300c9b0a8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.131147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.131006 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9m8t\" (UniqueName: \"kubernetes.io/projected/54384483-f540-422a-bb19-617300c9b0a8-kube-api-access-w9m8t\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.131147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.131097 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.131352 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:07:21.131242 4431 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-serving-cert: secret "maas-api-e2e-worker-w1-8d44dd-serving-cert" not found Oct 30 06:07:21.131352 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:07:21.131328 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls podName:54384483-f540-422a-bb19-617300c9b0a8 nodeName:}" failed. No retries permitted until 2026-10-30 06:07:21.631292183 +0000 UTC m=+850.437813912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls") pod "maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" (UID: "54384483-f540-422a-bb19-617300c9b0a8") : secret "maas-api-e2e-worker-w1-8d44dd-serving-cert" not found Oct 30 06:07:21.132722 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.132696 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54384483-f540-422a-bb19-617300c9b0a8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.134412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.134387 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-metrics-certs\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.158988 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.158961 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9m8t\" (UniqueName: \"kubernetes.io/projected/54384483-f540-422a-bb19-617300c9b0a8-kube-api-access-w9m8t\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.637600 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.637565 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.640006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.639972 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:21.831165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:21.831127 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:22.138888 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:22.138834 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp"] Oct 30 06:07:22.140562 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:07:22.140531 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54384483_f540_422a_bb19_617300c9b0a8.slice/crio-2f16ad1fe4cfdca7a377b3ea1adc01c57637e1789a43afc9b48461c44d60e40e WatchSource:0}: Error finding container 2f16ad1fe4cfdca7a377b3ea1adc01c57637e1789a43afc9b48461c44d60e40e: Status 404 returned error can't find the container with id 2f16ad1fe4cfdca7a377b3ea1adc01c57637e1789a43afc9b48461c44d60e40e Oct 30 06:07:22.770191 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:22.770143 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" event={"ID":"54384483-f540-422a-bb19-617300c9b0a8","Type":"ContainerStarted","Data":"522c2500daef39a312dd4122a56a3158806a7e43783c45a6800322963ac97ad7"} Oct 30 06:07:22.770405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:22.770219 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" event={"ID":"54384483-f540-422a-bb19-617300c9b0a8","Type":"ContainerStarted","Data":"2f16ad1fe4cfdca7a377b3ea1adc01c57637e1789a43afc9b48461c44d60e40e"} Oct 30 06:07:22.770405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:22.770330 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:22.820798 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:22.820698 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" podStartSLOduration=2.420852177 podStartE2EDuration="2.820674541s" totalImagesPullingTime="399.822364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:22.818923833 +0000 UTC m=+851.625445581" watchObservedRunningTime="2026-10-30 06:07:22.820674541 +0000 UTC m=+851.627196290" Oct 30 06:07:23.775845 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:23.775776 4431 generic.go:356] "Generic (PLEG): container finished" podID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerID="a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48" exitCode=0 Oct 30 06:07:23.776240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:23.775847 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" event={"ID":"84beb3ad-bff6-4552-b721-dc7a05b4cc92","Type":"ContainerDied","Data":"a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48"} Oct 30 06:07:23.776287 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:23.776255 4431 scope.go:122] "RemoveContainer" containerID="a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48" Oct 30 06:07:23.777612 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:23.777583 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:07:23.777725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:23.777666 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:07:23.777725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:23.777715 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:07:24.798899 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:24.798854 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" event={"ID":"84beb3ad-bff6-4552-b721-dc7a05b4cc92","Type":"ContainerStarted","Data":"33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0"} Oct 30 06:07:24.929426 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:24.929392 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:07:24.930566 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:24.930539 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 06:07:24.930685 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:24.930593 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 06:07:25.354774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:25.354741 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:25.392338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:25.392281 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:25.815582 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:25.815548 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 06:07:25.930317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:25.930280 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 06:07:25.930455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:25.930352 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 06:07:26.930106 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:26.930076 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 06:07:26.930504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:26.930133 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 06:07:27.603945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:27.603910 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:07:27.930375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:27.930287 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 06:07:27.930375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:27.930347 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 06:07:28.787257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:28.787229 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 06:07:28.929338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:28.929286 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 06:07:28.929495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:28.929360 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 06:07:29.929779 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:29.929745 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 06:07:29.930161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:29.929802 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 06:07:30.934143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:30.934112 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:07:30.935211 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:30.935188 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:07:33.701439 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:07:33.701399 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:07:44.179656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.179617 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6"] Oct 30 06:07:44.184743 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.184721 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.226659 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.226629 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"premium-w3-8d6349-kserve-self-signed-certs\"" Oct 30 06:07:44.229938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.229908 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6"] Oct 30 06:07:44.280765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.280731 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-home\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.280907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.280799 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25l6h\" (UniqueName: \"kubernetes.io/projected/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-kube-api-access-25l6h\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.280957 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.280907 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tmp-dir\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.281014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.280962 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-model-cache\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.281014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.280980 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-dshm\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.281127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.281108 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tls-certs\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.382699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.382649 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tmp-dir\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.382908 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.382748 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-model-cache\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.382908 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.382797 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-dshm\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.383066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.383042 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tls-certs\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.383286 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.383270 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-home\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.383499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.383473 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l6h\" (UniqueName: \"kubernetes.io/projected/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-kube-api-access-25l6h\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.383677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.383659 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tmp-dir\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.383782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.383726 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-home\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.384036 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.384013 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-model-cache\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.385529 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.385501 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-dshm\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.386098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.386081 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tls-certs\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.406365 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.406344 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25l6h\" (UniqueName: \"kubernetes.io/projected/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-kube-api-access-25l6h\") pod \"premium-w3-8d6349-kserve-6bb7bd68c-xhlb6\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.532122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.532095 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:44.832878 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.832847 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6"] Oct 30 06:07:44.835288 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:07:44.835262 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd64c38d6_979d_4e74_b2ea_ba1d34ccecaf.slice/crio-d86c5adc5178363eaed917ea26701cdbfd806f03c3d6f8ff6715f1dad2a3eff1 WatchSource:0}: Error finding container d86c5adc5178363eaed917ea26701cdbfd806f03c3d6f8ff6715f1dad2a3eff1: Status 404 returned error can't find the container with id d86c5adc5178363eaed917ea26701cdbfd806f03c3d6f8ff6715f1dad2a3eff1 Oct 30 06:07:44.902474 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.902446 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" event={"ID":"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf","Type":"ContainerStarted","Data":"c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4"} Oct 30 06:07:44.902589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.902485 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" event={"ID":"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf","Type":"ContainerStarted","Data":"d86c5adc5178363eaed917ea26701cdbfd806f03c3d6f8ff6715f1dad2a3eff1"} Oct 30 06:07:44.944995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:44.944945 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" podStartSLOduration=0.944928076 podStartE2EDuration="944.928076ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:44.944161095 +0000 UTC m=+873.750682845" watchObservedRunningTime="2026-10-30 06:07:44.944928076 +0000 UTC m=+873.751449831" Oct 30 06:07:45.532900 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:45.532853 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:54.533287 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:54.533248 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:54.546039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:54.546016 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:54.962989 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:54.962962 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 06:07:57.145118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.145086 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8"] Oct 30 06:07:57.149757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.149734 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.177197 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.177176 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"premium-w2-aaf467-kserve-self-signed-certs\"" Oct 30 06:07:57.177347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.177279 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"kube-root-ca.crt\"" Oct 30 06:07:57.177539 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.177522 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"openshift-service-ca.crt\"" Oct 30 06:07:57.177604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.177542 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"default-dockercfg-sjppv\"" Oct 30 06:07:57.182421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.182401 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8"] Oct 30 06:07:57.314390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.314354 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-model-cache\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.314545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.314396 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-home\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.314545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.314489 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8xb\" (UniqueName: \"kubernetes.io/projected/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-kube-api-access-ph8xb\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.314545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.314533 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-dshm\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.314672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.314627 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tmp-dir\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.314727 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.314684 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tls-certs\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.416039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.415951 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-model-cache\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.416039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.415986 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-home\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.416039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.416008 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8xb\" (UniqueName: \"kubernetes.io/projected/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-kube-api-access-ph8xb\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.416039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.416033 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-dshm\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.416402 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.416080 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tmp-dir\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.416402 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.416134 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tls-certs\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.416516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.416424 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-model-cache\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.416516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.416491 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-home\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.416611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.416515 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tmp-dir\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.418426 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.418396 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-dshm\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.418669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.418650 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tls-certs\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.438490 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.438470 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8xb\" (UniqueName: \"kubernetes.io/projected/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-kube-api-access-ph8xb\") pod \"premium-w2-aaf467-kserve-d8d84c77c-v6bn8\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.463352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.463326 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:07:57.789840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.789814 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8"] Oct 30 06:07:57.792190 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:07:57.792160 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f0c53ac_dd7d_4ee0_b707_51857ccf2724.slice/crio-69ed716df3beac64fb6c8ae446e92e7b15579a4575d427d3f716da579fb43892 WatchSource:0}: Error finding container 69ed716df3beac64fb6c8ae446e92e7b15579a4575d427d3f716da579fb43892: Status 404 returned error can't find the container with id 69ed716df3beac64fb6c8ae446e92e7b15579a4575d427d3f716da579fb43892 Oct 30 06:07:57.967745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.967709 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" event={"ID":"0f0c53ac-dd7d-4ee0-b707-51857ccf2724","Type":"ContainerStarted","Data":"2a591ad2c50a1e4154ffb4fcd6c747d0b8e70cf1321ab4d81d2692974fa6b5ab"} Oct 30 06:07:57.967887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:57.967755 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" event={"ID":"0f0c53ac-dd7d-4ee0-b707-51857ccf2724","Type":"ContainerStarted","Data":"69ed716df3beac64fb6c8ae446e92e7b15579a4575d427d3f716da579fb43892"} Oct 30 06:07:58.008677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:58.008607 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" podStartSLOduration=1.008592947 podStartE2EDuration="1.008592947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:58.008257374 +0000 UTC m=+886.814779123" watchObservedRunningTime="2026-10-30 06:07:58.008592947 +0000 UTC m=+886.815114695" Oct 30 06:07:58.464018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:07:58.463989 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:08:03.702653 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:08:03.702619 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:08:07.464186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:07.464148 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:08:07.476833 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:07.476807 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:08:08.028620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:08.028592 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 06:08:26.638545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.638508 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524"] Oct 30 06:08:26.644047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.644025 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.672111 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.672077 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"kube-root-ca.crt\"" Oct 30 06:08:26.672249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.672200 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"default-dockercfg-jbw59\"" Oct 30 06:08:26.672334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.672244 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"openshift-service-ca.crt\"" Oct 30 06:08:26.672334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.672324 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"premium-w1-8d44dd-kserve-self-signed-certs\"" Oct 30 06:08:26.675531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.675508 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524"] Oct 30 06:08:26.796736 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.796694 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9jv2\" (UniqueName: \"kubernetes.io/projected/713a2758-f5d8-4a1d-a9b3-3331530ead0a-kube-api-access-q9jv2\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.796924 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.796746 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-model-cache\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.796924 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.796867 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tls-certs\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.797002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.796931 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-home\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.797037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.797010 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-dshm\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.797074 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.797044 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tmp-dir\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.898568 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.898462 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9jv2\" (UniqueName: \"kubernetes.io/projected/713a2758-f5d8-4a1d-a9b3-3331530ead0a-kube-api-access-q9jv2\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.898568 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.898525 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-model-cache\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.898815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.898597 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tls-certs\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.898815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.898659 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-home\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.898815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.898703 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-dshm\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.898815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.898730 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tmp-dir\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.899064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.899045 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-home\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.899131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.899080 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-model-cache\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.899193 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.899145 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tmp-dir\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.901052 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.901031 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-dshm\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.901202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.901186 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tls-certs\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.937951 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.937912 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9jv2\" (UniqueName: \"kubernetes.io/projected/713a2758-f5d8-4a1d-a9b3-3331530ead0a-kube-api-access-q9jv2\") pod \"premium-w1-8d44dd-kserve-5ff97bbffc-6x524\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:26.956031 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:26.955999 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:27.279250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:27.279226 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524"] Oct 30 06:08:27.281513 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:08:27.281490 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod713a2758_f5d8_4a1d_a9b3_3331530ead0a.slice/crio-39455ff6986350aee8105ccf65ed593f767564a91bbe8dffa2bf35d5b5da204a WatchSource:0}: Error finding container 39455ff6986350aee8105ccf65ed593f767564a91bbe8dffa2bf35d5b5da204a: Status 404 returned error can't find the container with id 39455ff6986350aee8105ccf65ed593f767564a91bbe8dffa2bf35d5b5da204a Oct 30 06:08:28.119606 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:28.119564 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" event={"ID":"713a2758-f5d8-4a1d-a9b3-3331530ead0a","Type":"ContainerStarted","Data":"ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5"} Oct 30 06:08:28.119986 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:28.119614 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" event={"ID":"713a2758-f5d8-4a1d-a9b3-3331530ead0a","Type":"ContainerStarted","Data":"39455ff6986350aee8105ccf65ed593f767564a91bbe8dffa2bf35d5b5da204a"} Oct 30 06:08:28.160511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:28.160460 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" podStartSLOduration=2.160446163 podStartE2EDuration="2.160446163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:28.160007507 +0000 UTC m=+916.966529249" watchObservedRunningTime="2026-10-30 06:08:28.160446163 +0000 UTC m=+916.966967911" Oct 30 06:08:28.956613 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:28.956580 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:33.700101 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:08:33.700058 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:08:36.956232 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:36.956194 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:36.969717 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:36.969686 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:08:37.176526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:08:37.176493 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 06:09:03.700657 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:09:03.700567 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:09:18.635232 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.635186 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw"] Oct 30 06:09:18.639756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.639730 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.655839 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.655817 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw"] Oct 30 06:09:18.665842 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.665819 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"e2e-conflict-a-ce7e342c-kserve-self-signed-certs\"" Oct 30 06:09:18.703929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.703901 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-dshm\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.704029 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.703956 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-tmp-dir\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.704070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.704023 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskgc\" (UniqueName: \"kubernetes.io/projected/25696b68-c955-4e24-9f98-be083004ecd8-kube-api-access-tskgc\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.704070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.704063 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-model-cache\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.704177 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.704165 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-home\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.704211 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.704181 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25696b68-c955-4e24-9f98-be083004ecd8-tls-certs\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.804926 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.804896 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-dshm\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.805037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.804993 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-tmp-dir\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.805037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.805031 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskgc\" (UniqueName: \"kubernetes.io/projected/25696b68-c955-4e24-9f98-be083004ecd8-kube-api-access-tskgc\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.805136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.805070 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-model-cache\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.805186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.805136 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-home\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.805186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.805160 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25696b68-c955-4e24-9f98-be083004ecd8-tls-certs\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.805458 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.805433 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-tmp-dir\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.805636 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.805615 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-model-cache\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.805702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.805656 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-home\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.807764 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.807741 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-dshm\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.808082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.808044 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25696b68-c955-4e24-9f98-be083004ecd8-tls-certs\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.829059 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.829024 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskgc\" (UniqueName: \"kubernetes.io/projected/25696b68-c955-4e24-9f98-be083004ecd8-kube-api-access-tskgc\") pod \"e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:18.951840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:18.951813 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:19.036327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.036008 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh"] Oct 30 06:09:19.041576 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.041556 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.059100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.058496 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8d6349-openshift-default-dockercfg-xs2qm\"" Oct 30 06:09:19.061574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.061548 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh"] Oct 30 06:09:19.107724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.107695 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.107855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.107741 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.107855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.107766 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts9kn\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-kube-api-access-ts9kn\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.107937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.107901 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.107937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.107931 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.108001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.107989 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.108037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.108025 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.108071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.108041 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.108109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.108092 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.208650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208573 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.208650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208645 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.208844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208684 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts9kn\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-kube-api-access-ts9kn\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.208844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208732 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.208844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208768 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.208844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208827 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.209089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208864 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.209089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208926 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.209089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208959 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.209089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.208978 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.209089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.209053 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.209389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.209367 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.209502 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.209479 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.209635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.209618 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.211047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.211029 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.211444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.211427 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.231969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.231948 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts9kn\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-kube-api-access-ts9kn\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.232724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.232706 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.251590 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.251565 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw"] Oct 30 06:09:19.253643 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:09:19.253617 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25696b68_c955_4e24_9f98_be083004ecd8.slice/crio-7d7f60f456431fe53b5756001c994e846b8d45f4d2a9f9d39abb9f74413f55c0 WatchSource:0}: Error finding container 7d7f60f456431fe53b5756001c994e846b8d45f4d2a9f9d39abb9f74413f55c0: Status 404 returned error can't find the container with id 7d7f60f456431fe53b5756001c994e846b8d45f4d2a9f9d39abb9f74413f55c0 Oct 30 06:09:19.355449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.355423 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:19.381515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.381487 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" event={"ID":"25696b68-c955-4e24-9f98-be083004ecd8","Type":"ContainerStarted","Data":"650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2"} Oct 30 06:09:19.381648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.381534 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" event={"ID":"25696b68-c955-4e24-9f98-be083004ecd8","Type":"ContainerStarted","Data":"7d7f60f456431fe53b5756001c994e846b8d45f4d2a9f9d39abb9f74413f55c0"} Oct 30 06:09:19.424176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.424130 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" podStartSLOduration=1.4241169679999999 podStartE2EDuration="1.424116968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:19.421525989 +0000 UTC m=+968.228047737" watchObservedRunningTime="2026-10-30 06:09:19.424116968 +0000 UTC m=+968.230638716" Oct 30 06:09:19.657433 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.657404 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh"] Oct 30 06:09:19.659382 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:09:19.659355 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f273d9_6d40_407b_ab96_0da3ea0f3582.slice/crio-1fe4906c862f5477a5153f6467a69022a313d4588496d2490a09df422129dbb5 WatchSource:0}: Error finding container 1fe4906c862f5477a5153f6467a69022a313d4588496d2490a09df422129dbb5: Status 404 returned error can't find the container with id 1fe4906c862f5477a5153f6467a69022a313d4588496d2490a09df422129dbb5 Oct 30 06:09:19.661586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.661554 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:09:19.661666 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.661616 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:09:19.661704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.661665 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:09:19.951891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:19.951865 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:20.390468 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:20.390372 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" event={"ID":"a6f273d9-6d40-407b-ab96-0da3ea0f3582","Type":"ContainerStarted","Data":"047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091"} Oct 30 06:09:20.390468 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:20.390444 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" event={"ID":"a6f273d9-6d40-407b-ab96-0da3ea0f3582","Type":"ContainerStarted","Data":"1fe4906c862f5477a5153f6467a69022a313d4588496d2490a09df422129dbb5"} Oct 30 06:09:20.458636 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:20.458590 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" podStartSLOduration=1.458577444 podStartE2EDuration="1.458577444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:20.445894055 +0000 UTC m=+969.252415813" watchObservedRunningTime="2026-10-30 06:09:20.458577444 +0000 UTC m=+969.265099192" Oct 30 06:09:21.355917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:21.355887 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:21.357451 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:21.357431 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:09:21.357507 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:21.357479 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 06:09:22.356363 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:22.356298 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:09:22.356769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:22.356397 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 06:09:23.355937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:23.355902 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:09:23.356112 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:23.355954 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 06:09:24.356527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:24.356493 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:09:24.356925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:24.356549 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 06:09:25.355818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:25.355788 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:09:25.355992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:25.355836 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 06:09:26.360003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:26.359974 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:26.360433 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:26.360412 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:26.361038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:26.361024 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:09:28.951953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:28.951926 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:28.970169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:28.970147 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:29.456182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:29.456154 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:09:33.700569 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:09:33.700534 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:09:34.549994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.549963 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6"] Oct 30 06:09:34.553728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.553712 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.572118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.572097 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-aaf467-openshift-default-dockercfg-txgqd\"" Oct 30 06:09:34.586774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.586745 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6"] Oct 30 06:09:34.698580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.698539 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.698745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.698600 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.698745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.698686 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.698745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.698702 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8505f718-e49c-4231-91f0-20dd4a1be595-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.698745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.698721 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.698745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.698743 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.698909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.698892 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqn8f\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-kube-api-access-nqn8f\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.698953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.698939 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.699024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.699013 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8505f718-e49c-4231-91f0-20dd4a1be595-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.800618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.800528 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqn8f\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-kube-api-access-nqn8f\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.800618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.800594 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.800635 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8505f718-e49c-4231-91f0-20dd4a1be595-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.800674 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.800734 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.800806 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.800833 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8505f718-e49c-4231-91f0-20dd4a1be595-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.800860 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.800897 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.801122 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.801190 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.801288 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.801341 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.801638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.801614 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8505f718-e49c-4231-91f0-20dd4a1be595-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.802973 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.802944 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.803318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.803275 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8505f718-e49c-4231-91f0-20dd4a1be595-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.823853 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.823832 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.824069 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.824048 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqn8f\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-kube-api-access-nqn8f\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:34.867648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:34.867618 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:35.189975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:35.189943 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6"] Oct 30 06:09:35.191057 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:09:35.191031 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8505f718_e49c_4231_91f0_20dd4a1be595.slice/crio-851495299d9ab66139bf4ebf8deb06226f1f2008a4c1ce200ca4043a177e4210 WatchSource:0}: Error finding container 851495299d9ab66139bf4ebf8deb06226f1f2008a4c1ce200ca4043a177e4210: Status 404 returned error can't find the container with id 851495299d9ab66139bf4ebf8deb06226f1f2008a4c1ce200ca4043a177e4210 Oct 30 06:09:35.193562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:35.193531 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:09:35.193662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:35.193605 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:09:35.193662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:35.193647 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:09:35.474556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:35.474518 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" event={"ID":"8505f718-e49c-4231-91f0-20dd4a1be595","Type":"ContainerStarted","Data":"0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012"} Oct 30 06:09:35.474759 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:35.474588 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" event={"ID":"8505f718-e49c-4231-91f0-20dd4a1be595","Type":"ContainerStarted","Data":"851495299d9ab66139bf4ebf8deb06226f1f2008a4c1ce200ca4043a177e4210"} Oct 30 06:09:35.519711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:35.519667 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" podStartSLOduration=1.519653768 podStartE2EDuration="1.519653768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:35.518438804 +0000 UTC m=+984.324960552" watchObservedRunningTime="2026-10-30 06:09:35.519653768 +0000 UTC m=+984.326175515" Oct 30 06:09:35.871499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:35.871404 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:36.886048 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:36.886014 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 06:09:36.886570 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:36.886092 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 06:09:37.883334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:37.882620 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 06:09:37.883334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:37.882686 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 06:09:38.870654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:38.870615 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 06:09:38.871241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:38.871211 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 06:09:39.884454 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:39.884417 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 06:09:39.884903 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:39.884484 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 06:09:40.878327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:40.877615 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 06:09:40.878327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:40.877674 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 06:09:41.917830 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:41.917790 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:41.919832 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:41.919810 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:09:41.920990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:09:41.920963 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:10:03.700190 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:10:03.700153 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:10:33.700410 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:10:33.700369 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:10:41.300073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:41.300037 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw"] Oct 30 06:10:56.319964 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.319924 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" podUID="25696b68-c955-4e24-9f98-be083004ecd8" containerName="main" containerID="cri-o://650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2" gracePeriod=45 Oct 30 06:10:56.446330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.441782 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" podUID="25696b68-c955-4e24-9f98-be083004ecd8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused" Oct 30 06:10:56.588343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.588295 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:10:56.737153 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.737115 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25696b68-c955-4e24-9f98-be083004ecd8-kube-api-access-tskgc\" (UniqueName: \"kubernetes.io/projected/25696b68-c955-4e24-9f98-be083004ecd8-kube-api-access-tskgc\") pod \"25696b68-c955-4e24-9f98-be083004ecd8\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " Oct 30 06:10:56.737354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.737255 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-dshm\") pod \"25696b68-c955-4e24-9f98-be083004ecd8\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " Oct 30 06:10:56.737439 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.737363 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25696b68-c955-4e24-9f98-be083004ecd8-tls-certs\" (UniqueName: \"kubernetes.io/secret/25696b68-c955-4e24-9f98-be083004ecd8-tls-certs\") pod \"25696b68-c955-4e24-9f98-be083004ecd8\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " Oct 30 06:10:56.737439 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.737426 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-home\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-home\") pod \"25696b68-c955-4e24-9f98-be083004ecd8\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " Oct 30 06:10:56.737627 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.737475 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-tmp-dir\") pod \"25696b68-c955-4e24-9f98-be083004ecd8\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " Oct 30 06:10:56.737627 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.737530 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-model-cache\") pod \"25696b68-c955-4e24-9f98-be083004ecd8\" (UID: \"25696b68-c955-4e24-9f98-be083004ecd8\") " Oct 30 06:10:56.737760 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.737700 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-home" pod "25696b68-c955-4e24-9f98-be083004ecd8" (UID: "25696b68-c955-4e24-9f98-be083004ecd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:56.737923 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.737894 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-model-cache" pod "25696b68-c955-4e24-9f98-be083004ecd8" (UID: "25696b68-c955-4e24-9f98-be083004ecd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:56.738147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.738125 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-tmp-dir" pod "25696b68-c955-4e24-9f98-be083004ecd8" (UID: "25696b68-c955-4e24-9f98-be083004ecd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:56.738601 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.738579 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:10:56.738697 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.738606 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:10:56.738697 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.738620 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:10:56.739604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.739574 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25696b68-c955-4e24-9f98-be083004ecd8-kube-api-access-tskgc" pod "25696b68-c955-4e24-9f98-be083004ecd8" (UID: "25696b68-c955-4e24-9f98-be083004ecd8"). InnerVolumeSpecName "kube-api-access-tskgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:56.739707 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.739605 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-dshm" pod "25696b68-c955-4e24-9f98-be083004ecd8" (UID: "25696b68-c955-4e24-9f98-be083004ecd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:56.739791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.739775 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25696b68-c955-4e24-9f98-be083004ecd8-tls-certs" pod "25696b68-c955-4e24-9f98-be083004ecd8" (UID: "25696b68-c955-4e24-9f98-be083004ecd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:56.840008 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.839920 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25696b68-c955-4e24-9f98-be083004ecd8-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:10:56.840008 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.839950 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25696b68-c955-4e24-9f98-be083004ecd8-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:10:56.840008 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.839963 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tskgc\" (UniqueName: \"kubernetes.io/projected/25696b68-c955-4e24-9f98-be083004ecd8-kube-api-access-tskgc\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:10:56.910038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.909999 4431 generic.go:356] "Generic (PLEG): container finished" podID="25696b68-c955-4e24-9f98-be083004ecd8" containerID="650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2" exitCode=0 Oct 30 06:10:56.910213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.910044 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" event={"ID":"25696b68-c955-4e24-9f98-be083004ecd8","Type":"ContainerDied","Data":"650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2"} Oct 30 06:10:56.910213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.910084 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" Oct 30 06:10:56.910213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.910108 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw" event={"ID":"25696b68-c955-4e24-9f98-be083004ecd8","Type":"ContainerDied","Data":"7d7f60f456431fe53b5756001c994e846b8d45f4d2a9f9d39abb9f74413f55c0"} Oct 30 06:10:56.910213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.910131 4431 scope.go:122] "RemoveContainer" containerID="650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2" Oct 30 06:10:56.921624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.921602 4431 scope.go:122] "RemoveContainer" containerID="650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2" Oct 30 06:10:56.921921 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:10:56.921903 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2\": container with ID starting with 650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2 not found: ID does not exist" containerID="650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2" Oct 30 06:10:56.921993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.921930 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2"} err="failed to get container status \"650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2\": rpc error: code = NotFound desc = could not find container \"650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2\": container with ID starting with 650bef8cf05e3995f53363efaa866c156ee27778c2cee1ceff416a3ede88f9e2 not found: ID does not exist" Oct 30 06:10:56.962369 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.962342 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw"] Oct 30 06:10:56.969079 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:56.969056 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/e2e-conflict-a-ce7e342c-kserve-5cbf97984c-gl8nw"] Oct 30 06:10:57.337780 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:10:57.337749 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25696b68-c955-4e24-9f98-be083004ecd8" path="/var/lib/kubelet/pods/25696b68-c955-4e24-9f98-be083004ecd8/volumes" Oct 30 06:11:01.673610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.673574 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f"] Oct 30 06:11:01.674342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.674318 4431 cpu_manager.go:414] "containerMap: removing container" podUID="25696b68-c955-4e24-9f98-be083004ecd8" containerName="main" Oct 30 06:11:01.674342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.674339 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25696b68-c955-4e24-9f98-be083004ecd8" containerName="main" Oct 30 06:11:01.674541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.674502 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="25696b68-c955-4e24-9f98-be083004ecd8" containerName="main" Oct 30 06:11:01.681454 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.681435 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.683050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.683023 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrct5\" (UniqueName: \"kubernetes.io/projected/2b161474-abc3-429d-9e13-a0fb49f95717-kube-api-access-nrct5\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.683157 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.683110 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-tmp-dir\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.683157 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.683138 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b161474-abc3-429d-9e13-a0fb49f95717-tls-certs\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.683285 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.683167 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-model-cache\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.683285 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.683221 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-home\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.683285 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.683244 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-dshm\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.700428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.700379 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f"] Oct 30 06:11:01.704946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.704924 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8d44dd\"/\"distinct-w1-8d44dd-kserve-self-signed-certs\"" Oct 30 06:11:01.784158 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.784130 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-home\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.784277 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.784164 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-dshm\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.784277 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.784223 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrct5\" (UniqueName: \"kubernetes.io/projected/2b161474-abc3-429d-9e13-a0fb49f95717-kube-api-access-nrct5\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.784401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.784381 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-tmp-dir\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.784450 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.784434 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b161474-abc3-429d-9e13-a0fb49f95717-tls-certs\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.784499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.784478 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-model-cache\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.784579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.784561 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-home\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.784751 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.784736 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-model-cache\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.784787 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.784731 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-tmp-dir\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.786383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.786361 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-dshm\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.786743 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.786720 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b161474-abc3-429d-9e13-a0fb49f95717-tls-certs\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:01.812534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:01.812514 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrct5\" (UniqueName: \"kubernetes.io/projected/2b161474-abc3-429d-9e13-a0fb49f95717-kube-api-access-nrct5\") pod \"distinct-w1-8d44dd-kserve-975465b7b-2425f\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:02.004242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:02.004208 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:02.322417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:02.322387 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f"] Oct 30 06:11:02.324347 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:11:02.324318 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b161474_abc3_429d_9e13_a0fb49f95717.slice/crio-345285ef61b22a981ba708c8965f21cd5f5fdcfe02ec297ec8ebdb3aa618aa9a WatchSource:0}: Error finding container 345285ef61b22a981ba708c8965f21cd5f5fdcfe02ec297ec8ebdb3aa618aa9a: Status 404 returned error can't find the container with id 345285ef61b22a981ba708c8965f21cd5f5fdcfe02ec297ec8ebdb3aa618aa9a Oct 30 06:11:02.947945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:02.947905 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" event={"ID":"2b161474-abc3-429d-9e13-a0fb49f95717","Type":"ContainerStarted","Data":"29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522"} Oct 30 06:11:02.947945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:02.947948 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" event={"ID":"2b161474-abc3-429d-9e13-a0fb49f95717","Type":"ContainerStarted","Data":"345285ef61b22a981ba708c8965f21cd5f5fdcfe02ec297ec8ebdb3aa618aa9a"} Oct 30 06:11:02.989444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:02.989396 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" podStartSLOduration=1.989382865 podStartE2EDuration="1.989382865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:02.988405368 +0000 UTC m=+1071.794927117" watchObservedRunningTime="2026-10-30 06:11:02.989382865 +0000 UTC m=+1071.795904612" Oct 30 06:11:03.004607 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:03.004580 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:03.701238 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:11:03.701208 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:11:04.417473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.417445 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr"] Oct 30 06:11:04.421286 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.421267 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.453589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.453555 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr"] Oct 30 06:11:04.509709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.509672 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.509904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.509734 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.509904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.509848 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.509904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.509888 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.510058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.509974 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.510058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.509997 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.510134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.510121 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c35553-4ed8-4baf-8106-e21aeb16eae4-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.510169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.510154 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.510218 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.510179 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-kube-api-access-k5g9c\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.610967 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611009 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611030 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611047 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611072 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611091 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611192 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c35553-4ed8-4baf-8106-e21aeb16eae4-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611228 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611599 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611253 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-kube-api-access-k5g9c\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611599 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611455 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611599 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611580 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611765 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.611930 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.611905 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.612063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.612045 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c35553-4ed8-4baf-8106-e21aeb16eae4-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.613486 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.613463 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.613644 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.613618 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.637369 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.637344 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-kube-api-access-k5g9c\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.637467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.637400 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:04.735460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:04.735436 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:05.068357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:05.068261 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr"] Oct 30 06:11:05.071429 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:11:05.071397 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c35553_4ed8_4baf_8106_e21aeb16eae4.slice/crio-45c6260cda03f17a0bad1ec9345881836ee0bba4e759fd94bedb64368de84192 WatchSource:0}: Error finding container 45c6260cda03f17a0bad1ec9345881836ee0bba4e759fd94bedb64368de84192: Status 404 returned error can't find the container with id 45c6260cda03f17a0bad1ec9345881836ee0bba4e759fd94bedb64368de84192 Oct 30 06:11:05.074080 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:05.074050 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:05.074161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:05.074116 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:05.074161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:05.074143 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:05.992988 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:05.992939 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" event={"ID":"50c35553-4ed8-4baf-8106-e21aeb16eae4","Type":"ContainerStarted","Data":"b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406"} Oct 30 06:11:05.993531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:05.993012 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" event={"ID":"50c35553-4ed8-4baf-8106-e21aeb16eae4","Type":"ContainerStarted","Data":"45c6260cda03f17a0bad1ec9345881836ee0bba4e759fd94bedb64368de84192"} Oct 30 06:11:06.736346 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:06.736285 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:06.737890 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:06.737869 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:06.737966 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:06.737925 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 06:11:07.735976 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:07.735938 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:07.736401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:07.735998 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 06:11:08.736016 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:08.735983 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:08.736416 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:08.736072 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 06:11:09.736469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:09.736435 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:09.736889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:09.736517 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 06:11:10.735860 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:10.735823 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:10.736040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:10.735892 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 06:11:11.735925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:11.735834 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:11.735925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:11.735900 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 06:11:12.004805 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.004710 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:12.018436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.018410 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:12.043415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.043390 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 06:11:12.066933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.066878 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" podStartSLOduration=8.06686519 podStartE2EDuration="8.06686519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:06.08462806 +0000 UTC m=+1074.891149808" watchObservedRunningTime="2026-10-30 06:11:12.06686519 +0000 UTC m=+1080.873386937" Oct 30 06:11:12.610959 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.610923 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6"] Oct 30 06:11:12.616738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.616711 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.633903 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.633877 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"kube-root-ca.crt\"" Oct 30 06:11:12.634052 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.633943 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"openshift-service-ca.crt\"" Oct 30 06:11:12.634120 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.634058 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"default-dockercfg-9mttb\"" Oct 30 06:11:12.637999 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.637973 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6"] Oct 30 06:11:12.645763 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.645740 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"distinct-w0-a519e9-kserve-self-signed-certs\"" Oct 30 06:11:12.716866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.716831 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-model-cache\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.716997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.716878 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-dshm\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.716997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.716984 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnlms\" (UniqueName: \"kubernetes.io/projected/eecc4014-0448-4d17-a646-acd40564b5cc-kube-api-access-nnlms\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.717074 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.717012 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eecc4014-0448-4d17-a646-acd40564b5cc-tls-certs\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.717074 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.717050 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-tmp-dir\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.717181 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.717162 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-home\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.740247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.740219 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:12.740646 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.740630 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:12.741390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.741368 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 06:11:12.818676 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.818650 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-model-cache\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.818676 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.818678 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-dshm\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.818908 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.818739 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnlms\" (UniqueName: \"kubernetes.io/projected/eecc4014-0448-4d17-a646-acd40564b5cc-kube-api-access-nnlms\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.818908 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.818760 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eecc4014-0448-4d17-a646-acd40564b5cc-tls-certs\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.819077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.819049 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-tmp-dir\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.819148 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.819080 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-model-cache\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.819215 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.819166 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-home\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.819379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.819358 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-tmp-dir\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.819534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.819513 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-home\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.820998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.820965 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-dshm\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.821357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.821338 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eecc4014-0448-4d17-a646-acd40564b5cc-tls-certs\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.844457 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.844428 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnlms\" (UniqueName: \"kubernetes.io/projected/eecc4014-0448-4d17-a646-acd40564b5cc-kube-api-access-nnlms\") pod \"distinct-w0-a519e9-kserve-7488d69c8c-54xr6\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:12.930337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:12.929619 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:13.277548 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:13.277378 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6"] Oct 30 06:11:13.279458 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:11:13.279413 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeecc4014_0448_4d17_a646_acd40564b5cc.slice/crio-f4bbb698737819b426b9d59b1952cf48ec20c9b389baaaf9e7ae98782ca82217 WatchSource:0}: Error finding container f4bbb698737819b426b9d59b1952cf48ec20c9b389baaaf9e7ae98782ca82217: Status 404 returned error can't find the container with id f4bbb698737819b426b9d59b1952cf48ec20c9b389baaaf9e7ae98782ca82217 Oct 30 06:11:14.123201 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:14.123160 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" event={"ID":"eecc4014-0448-4d17-a646-acd40564b5cc","Type":"ContainerStarted","Data":"0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6"} Oct 30 06:11:14.123761 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:14.123213 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" event={"ID":"eecc4014-0448-4d17-a646-acd40564b5cc","Type":"ContainerStarted","Data":"f4bbb698737819b426b9d59b1952cf48ec20c9b389baaaf9e7ae98782ca82217"} Oct 30 06:11:14.175642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:14.175530 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" podStartSLOduration=2.175511993 podStartE2EDuration="2.175511993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:14.172103877 +0000 UTC m=+1082.978625630" watchObservedRunningTime="2026-10-30 06:11:14.175511993 +0000 UTC m=+1082.982033744" Oct 30 06:11:14.930044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:14.930013 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:22.785891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.785856 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw"] Oct 30 06:11:22.789748 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.789726 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.805230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.805207 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8d44dd-openshift-default-dockercfg-nzqwp\"" Oct 30 06:11:22.819428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.819402 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw"] Oct 30 06:11:22.930220 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.930187 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:22.939989 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.939956 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzs5\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-kube-api-access-pwzs5\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.940144 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.940005 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.940144 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.940092 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39553a-daaf-482c-8be9-476826b80ffe-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.940144 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.940126 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.940283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.940153 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.940283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.940275 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.940362 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.940338 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39553a-daaf-482c-8be9-476826b80ffe-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.940398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.940383 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.940500 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.940487 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:22.943195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:22.943177 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:23.041545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.041466 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.041545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.041514 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.041545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.041537 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39553a-daaf-482c-8be9-476826b80ffe-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.041961 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.041577 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.041961 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.041639 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.041961 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.041805 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzs5\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-kube-api-access-pwzs5\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.042131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.041972 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.042131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.042008 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39553a-daaf-482c-8be9-476826b80ffe-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.042131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.042041 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.042277 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.042126 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.042366 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.042345 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39553a-daaf-482c-8be9-476826b80ffe-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.042571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.042547 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.042694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.042597 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.042694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.042618 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.044030 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.044009 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.044818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.044792 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39553a-daaf-482c-8be9-476826b80ffe-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.065262 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.065241 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.066752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.066733 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzs5\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-kube-api-access-pwzs5\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.102491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.102469 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:23.193035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.193002 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:11:23.423317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.423281 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw"] Oct 30 06:11:23.424371 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:11:23.424349 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc39553a_daaf_482c_8be9_476826b80ffe.slice/crio-addda89392a9f3b777500c9fe141fcec37e0d6aca7e314ddbb1bc4fa09feba8c WatchSource:0}: Error finding container addda89392a9f3b777500c9fe141fcec37e0d6aca7e314ddbb1bc4fa09feba8c: Status 404 returned error can't find the container with id addda89392a9f3b777500c9fe141fcec37e0d6aca7e314ddbb1bc4fa09feba8c Oct 30 06:11:23.426684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.426653 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:23.426801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.426728 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:23.426801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:23.426773 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:24.186962 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:24.186931 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" event={"ID":"fc39553a-daaf-482c-8be9-476826b80ffe","Type":"ContainerStarted","Data":"86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2"} Oct 30 06:11:24.187417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:24.186983 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" event={"ID":"fc39553a-daaf-482c-8be9-476826b80ffe","Type":"ContainerStarted","Data":"addda89392a9f3b777500c9fe141fcec37e0d6aca7e314ddbb1bc4fa09feba8c"} Oct 30 06:11:24.256488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:24.256437 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" podStartSLOduration=2.256423381 podStartE2EDuration="2.256423381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:24.242582608 +0000 UTC m=+1093.049104356" watchObservedRunningTime="2026-10-30 06:11:24.256423381 +0000 UTC m=+1093.062945128" Oct 30 06:11:25.103506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:25.103471 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:25.105013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:25.104989 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 06:11:25.105076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:25.105040 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 06:11:26.109325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:26.108603 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 06:11:26.109325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:26.108680 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 06:11:27.104725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:27.104684 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 06:11:27.105066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:27.105032 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 06:11:28.103051 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:28.103013 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 06:11:28.103483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:28.103068 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 06:11:29.103436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:29.103399 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 06:11:29.103847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:29.103462 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 06:11:30.102939 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:30.102902 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 06:11:30.103119 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:30.102953 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 06:11:31.107559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:31.107520 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:31.107975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:31.107856 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:31.108525 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:31.108505 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 06:11:33.701058 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:11:33.701022 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:11:34.929004 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:34.928971 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg"] Oct 30 06:11:34.932978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:34.932959 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:34.966321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:34.966279 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg"] Oct 30 06:11:35.003720 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.003689 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.003720 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.003721 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.003920 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.003756 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj642\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-kube-api-access-zj642\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.003981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.003919 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.004061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.004044 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.004105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.004075 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.004180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.004163 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.004242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.004219 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.004343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.004330 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105193 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj642\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-kube-api-access-zj642\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105388 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105255 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105388 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105374 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105408 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105493 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105526 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105590 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105742 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105641 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105742 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105667 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.105886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.105859 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.106035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.106011 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.106035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.106025 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.106157 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.106099 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.106217 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.106205 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.107666 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.107643 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.107884 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.107867 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.131037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.131014 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.131285 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.131267 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj642\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-kube-api-access-zj642\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.247063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.247031 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:35.566867 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.566845 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg"] Oct 30 06:11:35.567736 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:11:35.567711 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16757c77_6f4b_4447_abcc_4c5cbf3d27cc.slice/crio-00bc6a1c708ed2f2dc34d8333fdbd5448aab3070ef46e90a25afc50d9e7ced84 WatchSource:0}: Error finding container 00bc6a1c708ed2f2dc34d8333fdbd5448aab3070ef46e90a25afc50d9e7ced84: Status 404 returned error can't find the container with id 00bc6a1c708ed2f2dc34d8333fdbd5448aab3070ef46e90a25afc50d9e7ced84 Oct 30 06:11:35.570061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.570029 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:35.570143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.570094 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:35.570143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:35.570126 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:36.290035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:36.289999 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" event={"ID":"16757c77-6f4b-4447-abcc-4c5cbf3d27cc","Type":"ContainerStarted","Data":"bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed"} Oct 30 06:11:36.290478 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:36.290055 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" event={"ID":"16757c77-6f4b-4447-abcc-4c5cbf3d27cc","Type":"ContainerStarted","Data":"00bc6a1c708ed2f2dc34d8333fdbd5448aab3070ef46e90a25afc50d9e7ced84"} Oct 30 06:11:36.347682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:36.347630 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" podStartSLOduration=2.3476172 podStartE2EDuration="2.3476172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:36.346597319 +0000 UTC m=+1105.153119067" watchObservedRunningTime="2026-10-30 06:11:36.3476172 +0000 UTC m=+1105.154139008" Oct 30 06:11:37.247792 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:37.247757 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:37.249500 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:37.249472 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:37.249651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:37.249530 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 06:11:38.247910 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:38.247881 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:38.248490 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:38.248452 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 06:11:39.247983 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:39.247946 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:39.248453 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:39.248001 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 06:11:40.248390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:40.248343 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:40.248847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:40.248410 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 06:11:41.248182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:41.248153 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:41.248469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:41.248447 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 06:11:42.247994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:42.247958 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:42.248154 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:42.248033 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 06:11:43.251624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:43.251594 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:43.251998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:43.251874 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:43.252588 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:43.252574 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:11:49.983398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:49.983367 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt"] Oct 30 06:11:49.988417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:49.988395 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.008667 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.008645 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt"] Oct 30 06:11:50.017910 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.017880 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.018027 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.018009 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4kvs\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-kube-api-access-b4kvs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.018133 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.018111 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.018195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.018173 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.018255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.018242 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.018340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.018326 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.018400 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.018369 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.018514 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.018490 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.018570 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.018529 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5650c0a7-778e-4156-af63-0ddd3c37667c-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.119447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119411 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvs\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-kube-api-access-b4kvs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.119632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119479 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.119632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119500 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.119632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119538 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.119632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119595 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.119852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119653 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.119852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119723 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.119852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119752 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5650c0a7-778e-4156-af63-0ddd3c37667c-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.119852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119807 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.120052 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119963 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.120052 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.119991 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.120159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.120048 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.120159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.120092 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.120510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.120491 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5650c0a7-778e-4156-af63-0ddd3c37667c-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.122083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.122056 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.122180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.122125 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.152511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.152481 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.152647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.152631 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4kvs\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-kube-api-access-b4kvs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.302708 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.302633 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:50.621631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.621602 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt"] Oct 30 06:11:50.623311 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:11:50.623263 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5650c0a7_778e_4156_af63_0ddd3c37667c.slice/crio-6e0f99418c14889c8da6d8ea3491362416c5b65bae1dabdc344a4b2747ff5567 WatchSource:0}: Error finding container 6e0f99418c14889c8da6d8ea3491362416c5b65bae1dabdc344a4b2747ff5567: Status 404 returned error can't find the container with id 6e0f99418c14889c8da6d8ea3491362416c5b65bae1dabdc344a4b2747ff5567 Oct 30 06:11:50.626363 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.626287 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:50.626438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.626413 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:50.626511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:50.626455 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:51.391314 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:51.391262 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" event={"ID":"5650c0a7-778e-4156-af63-0ddd3c37667c","Type":"ContainerStarted","Data":"98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9"} Oct 30 06:11:51.391804 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:51.391352 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" event={"ID":"5650c0a7-778e-4156-af63-0ddd3c37667c","Type":"ContainerStarted","Data":"6e0f99418c14889c8da6d8ea3491362416c5b65bae1dabdc344a4b2747ff5567"} Oct 30 06:11:51.480315 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:51.460500 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" podStartSLOduration=2.460481983 podStartE2EDuration="2.460481983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:51.447364713 +0000 UTC m=+1120.253886462" watchObservedRunningTime="2026-10-30 06:11:51.460481983 +0000 UTC m=+1120.267003734" Oct 30 06:11:52.141416 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.141383 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f"] Oct 30 06:11:52.145145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.145127 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.166039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.164625 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-eaee88-openshift-default-dockercfg-fmzvg\"" Oct 30 06:11:52.201418 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.194095 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f"] Oct 30 06:11:52.250473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.250438 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.250626 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.250476 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.250626 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.250579 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.250700 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.250678 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.250737 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.250712 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.250737 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.250732 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.250827 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.250814 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.250879 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.250869 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac68ecd6-eac6-4c11-94ec-495b86717e56-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.250915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.250892 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdpv\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-kube-api-access-9sdpv\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.303780 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.303746 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:52.305108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.305081 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:52.305244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.305128 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 06:11:52.352395 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352369 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352521 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352401 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352521 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352456 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352521 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352511 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac68ecd6-eac6-4c11-94ec-495b86717e56-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352533 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdpv\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-kube-api-access-9sdpv\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352576 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352612 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352776 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352677 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352776 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352719 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352776 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352755 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.352963 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.352945 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.353597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.353546 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac68ecd6-eac6-4c11-94ec-495b86717e56-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.353597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.353590 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.353770 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.353678 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.355085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.355059 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.355261 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.355239 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.390333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.389522 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.390611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.390593 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdpv\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-kube-api-access-9sdpv\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.459067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.459041 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:52.813945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.813912 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f"] Oct 30 06:11:52.816398 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:11:52.816343 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac68ecd6_eac6_4c11_94ec_495b86717e56.slice/crio-526096733bbeb53512cbf8a8600eac5dc9d21a9003f6d5cc480b1326f8cbad03 WatchSource:0}: Error finding container 526096733bbeb53512cbf8a8600eac5dc9d21a9003f6d5cc480b1326f8cbad03: Status 404 returned error can't find the container with id 526096733bbeb53512cbf8a8600eac5dc9d21a9003f6d5cc480b1326f8cbad03 Oct 30 06:11:52.824442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.820330 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:52.824442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.820403 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:52.824442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:52.820442 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:11:53.303271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:53.303232 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:53.303531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:53.303509 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 06:11:53.410021 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:53.409973 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" event={"ID":"ac68ecd6-eac6-4c11-94ec-495b86717e56","Type":"ContainerStarted","Data":"663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54"} Oct 30 06:11:53.410211 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:53.410039 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" event={"ID":"ac68ecd6-eac6-4c11-94ec-495b86717e56","Type":"ContainerStarted","Data":"526096733bbeb53512cbf8a8600eac5dc9d21a9003f6d5cc480b1326f8cbad03"} Oct 30 06:11:53.509324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:53.508387 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:11:53.511142 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:53.510205 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" podStartSLOduration=1.510183977 podStartE2EDuration="1.510183977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:53.499646165 +0000 UTC m=+1122.306167914" watchObservedRunningTime="2026-10-30 06:11:53.510183977 +0000 UTC m=+1122.316705726" Oct 30 06:11:53.511142 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:53.510707 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:53.511142 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:53.510757 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 06:11:54.311371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:54.310837 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:54.311371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:54.310904 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 06:11:54.464334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:54.463663 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:54.464334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:54.463722 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 06:11:55.312447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:55.312417 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:55.312902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:55.312507 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 06:11:55.480790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:55.480737 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:55.481392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:55.481357 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 06:11:56.306716 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:56.306678 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:56.307000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:56.306973 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 06:11:56.471327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:56.459634 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:56.471921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:56.471883 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 06:11:57.303014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.302981 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:57.303247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.303033 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 06:11:57.459677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.459641 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:57.459962 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.459934 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 06:11:57.646406 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.646325 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf"] Oct 30 06:11:57.651190 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.651165 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.695712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.695683 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"distinct-2-w0-a519e9-kserve-self-signed-certs\"" Oct 30 06:11:57.697467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.697258 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf"] Oct 30 06:11:57.797862 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.797830 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-dshm\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.798035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.797922 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779qw\" (UniqueName: \"kubernetes.io/projected/1d337a85-2994-4119-8d76-c0bbaca18d5e-kube-api-access-779qw\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.798035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.798005 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d337a85-2994-4119-8d76-c0bbaca18d5e-tls-certs\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.798155 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.798129 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-model-cache\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.798222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.798205 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-tmp-dir\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.798359 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.798340 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-home\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.900345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.900239 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-dshm\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.900526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.900364 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779qw\" (UniqueName: \"kubernetes.io/projected/1d337a85-2994-4119-8d76-c0bbaca18d5e-kube-api-access-779qw\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.900526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.900401 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d337a85-2994-4119-8d76-c0bbaca18d5e-tls-certs\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.900526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.900490 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-model-cache\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.900699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.900533 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-tmp-dir\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.900699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.900581 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-home\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.901025 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.901000 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-home\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.901193 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.901166 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-model-cache\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.901407 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.901387 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-tmp-dir\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.904462 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.904437 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-dshm\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.906416 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.906397 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d337a85-2994-4119-8d76-c0bbaca18d5e-tls-certs\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:57.923916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:57.923881 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779qw\" (UniqueName: \"kubernetes.io/projected/1d337a85-2994-4119-8d76-c0bbaca18d5e-kube-api-access-779qw\") pod \"distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:58.021338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:58.014777 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:11:58.307980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:58.307952 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:58.308263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:58.308240 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:58.308975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:58.308954 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:11:58.401735 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:58.401713 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf"] Oct 30 06:11:58.403647 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:11:58.403619 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d337a85_2994_4119_8d76_c0bbaca18d5e.slice/crio-113eba858a06ae2bd849722602bfa425ea06f80b63dcd716cfa1648e5a4e88a3 WatchSource:0}: Error finding container 113eba858a06ae2bd849722602bfa425ea06f80b63dcd716cfa1648e5a4e88a3: Status 404 returned error can't find the container with id 113eba858a06ae2bd849722602bfa425ea06f80b63dcd716cfa1648e5a4e88a3 Oct 30 06:11:58.442261 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:58.442222 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" event={"ID":"1d337a85-2994-4119-8d76-c0bbaca18d5e","Type":"ContainerStarted","Data":"113eba858a06ae2bd849722602bfa425ea06f80b63dcd716cfa1648e5a4e88a3"} Oct 30 06:11:58.469434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:58.469409 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:58.469632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:58.469608 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 06:11:59.450113 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:59.450076 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" event={"ID":"1d337a85-2994-4119-8d76-c0bbaca18d5e","Type":"ContainerStarted","Data":"a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720"} Oct 30 06:11:59.459906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:59.459874 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:59.460039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:59.459933 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 06:11:59.495747 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:11:59.495698 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" podStartSLOduration=2.495683491 podStartE2EDuration="2.495683491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:59.493391057 +0000 UTC m=+1128.299912806" watchObservedRunningTime="2026-10-30 06:11:59.495683491 +0000 UTC m=+1128.302205239" Oct 30 06:12:00.016258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:00.016222 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:12:00.464107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:00.464079 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:12:00.464605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:00.464347 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:12:00.465003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:00.464984 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:12:03.900327 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:12:03.897536 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:12:04.623328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.618243 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667"] Oct 30 06:12:04.630323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.630279 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:04.648326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.645071 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667"] Oct 30 06:12:04.833334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.828151 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:04.833334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.828204 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:04.833334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.828319 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:04.833334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.828359 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:04.833334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.828439 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:04.833334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.828483 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:04.833334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.828545 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:04.833334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.828613 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:04.833334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:04.828670 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5spr\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-kube-api-access-r5spr\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.009664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.009626 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.015459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.015427 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5spr\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-kube-api-access-r5spr\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.015703 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.015679 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.015822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.015807 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.032325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.016012 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.032681 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.018506 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.032893 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.032857 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.033136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.033114 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.033345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.033327 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.033568 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.033549 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.079324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.035039 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.079324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.042575 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.079324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.042742 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.079324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.043611 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.079324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.043676 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" podUID="4199dace-66ce-4982-9ff6-629345b077ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:05.116877 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.116843 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.172323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.137897 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.172323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.145244 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5spr\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-kube-api-access-r5spr\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:05.194576 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.192655 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:05.971332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:05.914941 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:06.142333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.106392 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" podUID="0f0c53ac-dd7d-4ee0-b707-51857ccf2724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 06:12:06.302383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.224462 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" podUID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:06.302383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.225702 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 06:12:06.388356 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.252556 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:06.434333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.430817 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" podUID="d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 06:12:06.466706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.466661 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" podUID="4199dace-66ce-4982-9ff6-629345b077ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 06:12:06.714456 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.714406 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" podUID="80db6c09-1940-4274-a21c-57ab7dd6738c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:06.718931 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:12:06.718904 4431 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.387s" Oct 30 06:12:06.719137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.719122 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:12:06.719285 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.719272 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:12:06.842949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.842863 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:12:06.897330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:06.858377 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:12:07.320660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:07.318945 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667"] Oct 30 06:12:07.327905 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:07.327853 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:07.328024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:07.327945 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:07.328024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:07.327990 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:07.635863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:07.635760 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" event={"ID":"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4","Type":"ContainerStarted","Data":"0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804"} Oct 30 06:12:07.635863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:07.635830 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" event={"ID":"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4","Type":"ContainerStarted","Data":"1a89da679f178be84e6ccef023acc2012568a3ce9693fd975a992ad218b99595"} Oct 30 06:12:07.682443 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:07.682384 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" podStartSLOduration=3.6823635770000003 podStartE2EDuration="3.682363577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:07.681284338 +0000 UTC m=+1136.487806103" watchObservedRunningTime="2026-10-30 06:12:07.682363577 +0000 UTC m=+1136.488885326" Oct 30 06:12:08.016148 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:08.016115 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:12:08.029085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:08.029060 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:12:08.254629 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:08.254596 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:08.256095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:08.256067 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:12:08.256237 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:08.256126 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 06:12:08.653064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:08.653037 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:12:09.254869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:09.254830 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:12:09.255048 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:09.254888 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 06:12:10.254370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:10.254336 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:12:10.254765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:10.254388 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 06:12:11.254428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:11.254396 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:12:11.254907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:11.254449 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 06:12:12.255202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:12.255156 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:12:12.255624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:12.255246 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 06:12:13.255247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:13.255210 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:12:13.255664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:13.255263 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 06:12:14.758485 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:14.758444 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:14.759040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:14.759020 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:14.760605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:14.760580 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:12:18.632311 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.632266 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9"] Oct 30 06:12:18.637672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.637652 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:18.661932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.661905 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9"] Oct 30 06:12:18.669569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.669549 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1ec766-dockercfg-47w5p\"" Oct 30 06:12:18.669569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.669560 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1ec766\"" Oct 30 06:12:18.733239 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.733156 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace1f0f2-b07b-4698-affb-482ce2bf4a00-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9\" (UID: \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:18.733456 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.733380 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whhhg\" (UniqueName: \"kubernetes.io/projected/ace1f0f2-b07b-4698-affb-482ce2bf4a00-kube-api-access-whhhg\") pod \"payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9\" (UID: \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:18.834322 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.834273 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whhhg\" (UniqueName: \"kubernetes.io/projected/ace1f0f2-b07b-4698-affb-482ce2bf4a00-kube-api-access-whhhg\") pod \"payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9\" (UID: \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:18.834563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.834544 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace1f0f2-b07b-4698-affb-482ce2bf4a00-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9\" (UID: \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:18.835210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.835188 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace1f0f2-b07b-4698-affb-482ce2bf4a00-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9\" (UID: \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:18.857159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.857131 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whhhg\" (UniqueName: \"kubernetes.io/projected/ace1f0f2-b07b-4698-affb-482ce2bf4a00-kube-api-access-whhhg\") pod \"payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9\" (UID: \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:18.950979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:18.950949 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:19.258478 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:19.258452 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9"] Oct 30 06:12:19.259256 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:12:19.259228 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace1f0f2_b07b_4698_affb_482ce2bf4a00.slice/crio-c6e06b95c6bfcb200eb3fdcc549985bb2246d5e516547d9caa7f0ae044e0df90 WatchSource:0}: Error finding container c6e06b95c6bfcb200eb3fdcc549985bb2246d5e516547d9caa7f0ae044e0df90: Status 404 returned error can't find the container with id c6e06b95c6bfcb200eb3fdcc549985bb2246d5e516547d9caa7f0ae044e0df90 Oct 30 06:12:19.784005 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:19.783968 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" event={"ID":"ace1f0f2-b07b-4698-affb-482ce2bf4a00","Type":"ContainerStarted","Data":"1f242f6c01da9a730308bae9a019d036530c0e496cec381a6e63214a9b10b1c8"} Oct 30 06:12:19.784447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:19.784026 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" event={"ID":"ace1f0f2-b07b-4698-affb-482ce2bf4a00","Type":"ContainerStarted","Data":"c6e06b95c6bfcb200eb3fdcc549985bb2246d5e516547d9caa7f0ae044e0df90"} Oct 30 06:12:19.784447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:19.784059 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:19.824965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:19.824914 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" podStartSLOduration=1.8249015050000001 podStartE2EDuration="1.824901505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:19.823486585 +0000 UTC m=+1148.630008344" watchObservedRunningTime="2026-10-30 06:12:19.824901505 +0000 UTC m=+1148.631423303" Oct 30 06:12:30.584726 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.584687 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn"] Oct 30 06:12:30.588712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.588690 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.606814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.606788 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn"] Oct 30 06:12:30.615477 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.615449 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ba44b248\"/\"kube-root-ca.crt\"" Oct 30 06:12:30.615477 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.615468 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ba44b248\"/\"openshift-service-ca.crt\"" Oct 30 06:12:30.615639 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.615586 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ba44b248\"/\"e2e-lifecycle-model-ba44b248-kserve-self-signed-certs\"" Oct 30 06:12:30.618116 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.618095 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ba44b248\"/\"default-dockercfg-f4s56\"" Oct 30 06:12:30.722843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.722813 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-home\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.722843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.722843 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d99277-ebc7-448a-b6ca-a001b71eaa42-tls-certs\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.723019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.722871 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-tmp-dir\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.723077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.723057 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-dshm\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.723173 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.723158 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-model-cache\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.723213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.723184 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7f68\" (UniqueName: \"kubernetes.io/projected/43d99277-ebc7-448a-b6ca-a001b71eaa42-kube-api-access-x7f68\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.793180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.793154 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:12:30.824433 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.824399 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-model-cache\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.824703 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.824678 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7f68\" (UniqueName: \"kubernetes.io/projected/43d99277-ebc7-448a-b6ca-a001b71eaa42-kube-api-access-x7f68\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.824928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.824901 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-model-cache\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.824928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.824926 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-home\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.825118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.824957 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d99277-ebc7-448a-b6ca-a001b71eaa42-tls-certs\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.825118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.825005 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-tmp-dir\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.825229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.825148 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-dshm\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.825412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.825391 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-tmp-dir\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.827364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.827336 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-home\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.835984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.835919 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d99277-ebc7-448a-b6ca-a001b71eaa42-tls-certs\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.851512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.851477 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-dshm\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.855892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.855866 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7f68\" (UniqueName: \"kubernetes.io/projected/43d99277-ebc7-448a-b6ca-a001b71eaa42-kube-api-access-x7f68\") pod \"e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:30.902788 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:30.902753 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:31.213611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:31.213587 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn"] Oct 30 06:12:31.215416 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:12:31.215381 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43d99277_ebc7_448a_b6ca_a001b71eaa42.slice/crio-e5fc4b5f150062deb195428a9d92d750ebbdfabc4470eba68eee76411797d43a WatchSource:0}: Error finding container e5fc4b5f150062deb195428a9d92d750ebbdfabc4470eba68eee76411797d43a: Status 404 returned error can't find the container with id e5fc4b5f150062deb195428a9d92d750ebbdfabc4470eba68eee76411797d43a Oct 30 06:12:31.883832 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:31.883798 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" event={"ID":"43d99277-ebc7-448a-b6ca-a001b71eaa42","Type":"ContainerStarted","Data":"ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98"} Oct 30 06:12:31.883832 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:31.883837 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" event={"ID":"43d99277-ebc7-448a-b6ca-a001b71eaa42","Type":"ContainerStarted","Data":"e5fc4b5f150062deb195428a9d92d750ebbdfabc4470eba68eee76411797d43a"} Oct 30 06:12:31.902896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:31.902875 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:31.926637 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:31.926600 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" podStartSLOduration=1.926588378 podStartE2EDuration="1.926588378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:31.924238886 +0000 UTC m=+1160.730760636" watchObservedRunningTime="2026-10-30 06:12:31.926588378 +0000 UTC m=+1160.733110159" Oct 30 06:12:33.701322 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:12:33.701279 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:12:37.349283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.349249 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz"] Oct 30 06:12:37.353373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.353354 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.403938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.403914 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz"] Oct 30 06:12:37.404273 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.404252 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.404398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.404281 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.404398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.404377 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57acb25d-0b12-411c-831e-a7006448e7f5-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.404516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.404420 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.404516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.404439 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.404516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.404471 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5ls\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-kube-api-access-kg5ls\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.404696 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.404677 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.404752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.404706 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57acb25d-0b12-411c-831e-a7006448e7f5-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.404960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.404925 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.423769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.423742 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp"] Oct 30 06:12:37.434087 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.434058 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.465164 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.465137 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp"] Oct 30 06:12:37.506245 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506216 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.506387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506255 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5ls\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-kube-api-access-kg5ls\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.506387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506377 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.506480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506424 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.506480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506444 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57acb25d-0b12-411c-831e-a7006448e7f5-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.506586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506491 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.506586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506527 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.506687 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506588 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.506687 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506610 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.506687 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506666 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.506847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506717 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.506847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506794 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.506847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506817 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.506847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506834 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmj5n\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-kube-api-access-xmj5n\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.507034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506871 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.507034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.506973 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57acb25d-0b12-411c-831e-a7006448e7f5-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.507034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.507001 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.507196 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.507031 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.507196 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.507056 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.507196 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.507078 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.507196 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.507117 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57acb25d-0b12-411c-831e-a7006448e7f5-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.507377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.507191 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.507444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.507420 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.508811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.508790 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.509313 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.509282 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57acb25d-0b12-411c-831e-a7006448e7f5-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.529333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.529310 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5ls\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-kube-api-access-kg5ls\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.529830 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.529811 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.607819 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.607762 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.607819 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.607794 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.607993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.607829 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.607993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.607852 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmj5n\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-kube-api-access-xmj5n\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.607993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.607900 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.607993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.607933 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.608215 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.608003 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.608215 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.608064 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.608215 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.608108 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.608392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.608234 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.608535 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.608505 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.608640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.608550 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.608640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.608556 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.608640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.608582 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.610032 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.610011 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.610421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.610400 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.632598 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.632575 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.632678 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.632620 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmj5n\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-kube-api-access-xmj5n\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.667812 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.667790 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:37.754402 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.754195 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:37.989201 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.989176 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz"] Oct 30 06:12:37.990360 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:12:37.990333 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57acb25d_0b12_411c_831e_a7006448e7f5.slice/crio-26f5530cfe1fbeedd0f47fd3097b1905969d703dec90b73f8360a70d34dc86dc WatchSource:0}: Error finding container 26f5530cfe1fbeedd0f47fd3097b1905969d703dec90b73f8360a70d34dc86dc: Status 404 returned error can't find the container with id 26f5530cfe1fbeedd0f47fd3097b1905969d703dec90b73f8360a70d34dc86dc Oct 30 06:12:37.992698 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.992666 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:37.992774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.992736 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:37.992774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:37.992767 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:38.061620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:38.061597 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp"] Oct 30 06:12:38.062630 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:12:38.062576 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12e6cc5_e97d_4476_ba06_ab9d011c57b5.slice/crio-711aa2000b4bea9543ee935f3a9bc3c07cc6ed9d29302ddc63d46280d2c07ca5 WatchSource:0}: Error finding container 711aa2000b4bea9543ee935f3a9bc3c07cc6ed9d29302ddc63d46280d2c07ca5: Status 404 returned error can't find the container with id 711aa2000b4bea9543ee935f3a9bc3c07cc6ed9d29302ddc63d46280d2c07ca5 Oct 30 06:12:38.065373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:38.065342 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:38.065436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:38.065409 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:38.065483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:38.065442 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:38.930186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:38.930149 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" event={"ID":"b12e6cc5-e97d-4476-ba06-ab9d011c57b5","Type":"ContainerStarted","Data":"a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a"} Oct 30 06:12:38.930684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:38.930212 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" event={"ID":"b12e6cc5-e97d-4476-ba06-ab9d011c57b5","Type":"ContainerStarted","Data":"711aa2000b4bea9543ee935f3a9bc3c07cc6ed9d29302ddc63d46280d2c07ca5"} Oct 30 06:12:38.932162 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:38.932137 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" event={"ID":"57acb25d-0b12-411c-831e-a7006448e7f5","Type":"ContainerStarted","Data":"1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c"} Oct 30 06:12:38.932288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:38.932181 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" event={"ID":"57acb25d-0b12-411c-831e-a7006448e7f5","Type":"ContainerStarted","Data":"26f5530cfe1fbeedd0f47fd3097b1905969d703dec90b73f8360a70d34dc86dc"} Oct 30 06:12:38.991323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:38.990295 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" podStartSLOduration=1.990275526 podStartE2EDuration="1.990275526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:38.988807373 +0000 UTC m=+1167.795329132" watchObservedRunningTime="2026-10-30 06:12:38.990275526 +0000 UTC m=+1167.796797275" Oct 30 06:12:39.044109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:39.044053 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" podStartSLOduration=2.044039263 podStartE2EDuration="2.044039263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:39.042833276 +0000 UTC m=+1167.849355027" watchObservedRunningTime="2026-10-30 06:12:39.044039263 +0000 UTC m=+1167.850561011" Oct 30 06:12:39.670806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:39.670377 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:39.670806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:39.670493 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 06:12:39.670806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:39.670545 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 06:12:39.756320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:39.756280 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:39.756320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:39.756318 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:39.756510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:39.756363 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 06:12:40.675449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.675411 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 06:12:40.676368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.676333 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 06:12:40.754596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.754556 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:40.754811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.754626 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 06:12:40.845769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.845734 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c"] Oct 30 06:12:40.850076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.850054 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:40.931164 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.909434 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:40.931164 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.923449 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c"] Oct 30 06:12:40.945005 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.944978 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:40.955872 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.955841 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vz9g\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-kube-api-access-6vz9g\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:40.956106 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.956090 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:40.956272 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.956258 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:40.956406 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.956391 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:40.956513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.956501 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebc7419-b24e-42f5-9107-fe3143d88963-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:40.956659 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.956638 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:40.956723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.956678 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:40.956723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.956696 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:40.956824 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:40.956729 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebc7419-b24e-42f5-9107-fe3143d88963-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.016130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.016101 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:12:41.057550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.057503 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.057704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.057639 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.057704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.057689 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.057795 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.057781 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebc7419-b24e-42f5-9107-fe3143d88963-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.057891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.057875 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vz9g\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-kube-api-access-6vz9g\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.058044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.058026 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.058182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.058164 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.058255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.058234 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.058424 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.058411 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebc7419-b24e-42f5-9107-fe3143d88963-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.058596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.058574 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.058716 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.058694 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.058831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.058808 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.059039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.059015 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.071322 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.059740 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebc7419-b24e-42f5-9107-fe3143d88963-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.119357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.116031 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebc7419-b24e-42f5-9107-fe3143d88963-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.121635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.121602 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.122402 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.122355 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.123583 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.123559 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vz9g\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-kube-api-access-6vz9g\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.232974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.232944 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:41.613386 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.613352 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c"] Oct 30 06:12:41.614077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.614047 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:41.614158 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.614108 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:41.614158 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.614139 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:41.688324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.671739 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 06:12:41.688324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.671796 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 06:12:41.754991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.754966 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:41.755127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.755007 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 06:12:41.960328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.959246 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" event={"ID":"aebc7419-b24e-42f5-9107-fe3143d88963","Type":"ContainerStarted","Data":"192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e"} Oct 30 06:12:41.960328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:41.959292 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" event={"ID":"aebc7419-b24e-42f5-9107-fe3143d88963","Type":"ContainerStarted","Data":"98876fdb1f1eb71b8a28841f7b424ec81eb114f587e801b0d0131d070591f99b"} Oct 30 06:12:42.023322 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:42.022674 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podStartSLOduration=2.022653639 podStartE2EDuration="2.022653639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:42.012279542 +0000 UTC m=+1170.818801326" watchObservedRunningTime="2026-10-30 06:12:42.022653639 +0000 UTC m=+1170.829175389" Oct 30 06:12:42.237385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:42.237273 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:42.237554 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:42.237481 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 06:12:42.237627 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:42.237557 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 06:12:42.674511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:42.674417 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 06:12:42.674831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:42.674797 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 06:12:42.755154 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:42.755121 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:42.755623 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:42.755597 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 06:12:43.234326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:43.233699 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 06:12:43.234326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:43.233757 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 06:12:43.695324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:43.691521 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 06:12:43.695324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:43.691619 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 06:12:43.754868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:43.754820 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:43.755035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:43.754889 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 06:12:44.234313 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:44.234262 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 06:12:44.234780 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:44.234352 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 06:12:44.692384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:44.689678 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 06:12:44.692384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:44.689755 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 06:12:44.755412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:44.755380 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:44.755810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:44.755430 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 06:12:45.234024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:45.233985 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 06:12:45.234235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:45.234062 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 06:12:45.712719 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:45.712689 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:45.713360 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:45.713074 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:45.714978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:45.714956 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:12:45.769869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:45.768586 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:45.769869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:45.769707 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:45.783555 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:45.783524 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:12:46.233724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:46.233689 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 06:12:46.233899 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:46.233738 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 06:12:47.233922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:47.233868 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 06:12:47.234342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:47.233942 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 06:12:48.234397 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:48.234363 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 06:12:48.234790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:48.234433 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 06:12:49.245692 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:49.245660 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:49.246208 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:49.246151 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:49.247094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:49.247070 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:12:52.689993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.689961 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p"] Oct 30 06:12:52.695321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.695285 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.713881 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.713848 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 06:12:52.714059 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.713948 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 06:12:52.722326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.722285 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 06:12:52.722461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.722335 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dbg4t\"" Oct 30 06:12:52.727442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.727419 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p"] Oct 30 06:12:52.760678 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.760655 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.760793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.760720 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.760852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.760819 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.760903 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.760853 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.760995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.760976 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7q67\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-kube-api-access-d7q67\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.761109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.761093 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c07881-c3bc-48f0-aff4-a4f20e84f387-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.761185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.761169 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.862510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.862455 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.862689 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.862634 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.862777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.862742 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.862838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.862801 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.862904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.862835 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.862962 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.862912 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7q67\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-kube-api-access-d7q67\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.863086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.863027 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c07881-c3bc-48f0-aff4-a4f20e84f387-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.863485 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.863456 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.865055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.865034 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.865410 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.865387 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.865617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.865597 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c07881-c3bc-48f0-aff4-a4f20e84f387-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.865763 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.865746 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.885401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.885379 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:52.885600 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:52.885582 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7q67\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-kube-api-access-d7q67\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f5g5p\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:53.030032 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:53.025879 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:53.366723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:53.366685 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p"] Oct 30 06:12:53.371719 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:12:53.371690 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c07881_c3bc_48f0_aff4_a4f20e84f387.slice/crio-a6171871836c387681ad60e0bfb00ba61a30449cfe6fe1915bce9c803b728699 WatchSource:0}: Error finding container a6171871836c387681ad60e0bfb00ba61a30449cfe6fe1915bce9c803b728699: Status 404 returned error can't find the container with id a6171871836c387681ad60e0bfb00ba61a30449cfe6fe1915bce9c803b728699 Oct 30 06:12:53.374515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:53.374479 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:53.374668 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:53.374562 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:12:54.145846 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:54.145806 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" event={"ID":"71c07881-c3bc-48f0-aff4-a4f20e84f387","Type":"ContainerStarted","Data":"d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c"} Oct 30 06:12:54.146288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:54.145887 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" event={"ID":"71c07881-c3bc-48f0-aff4-a4f20e84f387","Type":"ContainerStarted","Data":"a6171871836c387681ad60e0bfb00ba61a30449cfe6fe1915bce9c803b728699"} Oct 30 06:12:54.146638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:54.146601 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:12:54.148393 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:54.148364 4431 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-f5g5p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:12:54.148493 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:54.148427 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" podUID="71c07881-c3bc-48f0-aff4-a4f20e84f387" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:12:54.197035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:54.196974 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" podStartSLOduration=2.196961495 podStartE2EDuration="2.196961495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:54.194447435 +0000 UTC m=+1183.000969175" watchObservedRunningTime="2026-10-30 06:12:54.196961495 +0000 UTC m=+1183.003483244" Oct 30 06:12:55.153435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:12:55.153408 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:13:03.701711 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:13:03.701676 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:13:25.986572 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:25.986522 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh"] Oct 30 06:13:25.992023 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:25.991999 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.014073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.014043 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh"] Oct 30 06:13:26.099767 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.099740 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.099945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.099804 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.099945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.099933 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.100049 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.099956 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.100049 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.099976 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.100049 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.100034 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ht8t\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-kube-api-access-8ht8t\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.100164 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.100104 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.100164 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.100143 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.100250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.100163 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201018 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201054 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201090 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201126 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ht8t\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-kube-api-access-8ht8t\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201165 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201211 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201239 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201489 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201520 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201776 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201860 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201823 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.201917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.201867 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.202132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.202104 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.203256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.203227 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.203995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.203973 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.207130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.207108 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.224540 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.224517 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ht8t\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-kube-api-access-8ht8t\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.225768 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.225747 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.305866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.305794 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:26.630828 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.630798 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh"] Oct 30 06:13:26.631562 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:13:26.631528 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e444187_d3e6_4f34_9a68_42eb1c6f7260.slice/crio-498ba6b2fe6bad246a46c140e1c2b13e663beee1598562cdcf3b6f1ad880cc44 WatchSource:0}: Error finding container 498ba6b2fe6bad246a46c140e1c2b13e663beee1598562cdcf3b6f1ad880cc44: Status 404 returned error can't find the container with id 498ba6b2fe6bad246a46c140e1c2b13e663beee1598562cdcf3b6f1ad880cc44 Oct 30 06:13:26.633885 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.633853 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:13:26.633975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.633918 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:13:26.633975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:26.633953 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:13:27.343251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:27.343215 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" event={"ID":"4e444187-d3e6-4f34-9a68-42eb1c6f7260","Type":"ContainerStarted","Data":"8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e"} Oct 30 06:13:27.343640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:27.343271 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" event={"ID":"4e444187-d3e6-4f34-9a68-42eb1c6f7260","Type":"ContainerStarted","Data":"498ba6b2fe6bad246a46c140e1c2b13e663beee1598562cdcf3b6f1ad880cc44"} Oct 30 06:13:27.414217 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:27.414165 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" podStartSLOduration=2.414151222 podStartE2EDuration="2.414151222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:27.400243081 +0000 UTC m=+1216.206764832" watchObservedRunningTime="2026-10-30 06:13:27.414151222 +0000 UTC m=+1216.220672970" Oct 30 06:13:28.307198 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:28.306848 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:28.309697 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:28.309662 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:13:28.310007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:28.309725 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 06:13:29.307124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:29.307086 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:13:29.307569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:29.307146 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 06:13:30.307175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:30.307137 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:13:30.307836 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:30.307805 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 06:13:31.306476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:31.306438 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:13:31.306687 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:31.306493 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 06:13:32.307188 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:32.307146 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:13:32.307873 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:32.307215 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 06:13:33.306926 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:33.306890 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:13:33.307123 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:33.306954 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 06:13:33.702109 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:13:33.702061 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:13:34.364903 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:34.364862 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:34.366097 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:34.366068 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:34.366806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:34.366775 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:13:46.376758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:46.376717 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9"] Oct 30 06:13:51.377733 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.377674 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" podUID="ace1f0f2-b07b-4698-affb-482ce2bf4a00" containerName="payload-pre-processing" containerID="cri-o://1f242f6c01da9a730308bae9a019d036530c0e496cec381a6e63214a9b10b1c8" gracePeriod=25 Oct 30 06:13:51.516920 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.516885 4431 generic.go:356] "Generic (PLEG): container finished" podID="ace1f0f2-b07b-4698-affb-482ce2bf4a00" containerID="1f242f6c01da9a730308bae9a019d036530c0e496cec381a6e63214a9b10b1c8" exitCode=0 Oct 30 06:13:51.517063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.516947 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" event={"ID":"ace1f0f2-b07b-4698-affb-482ce2bf4a00","Type":"ContainerDied","Data":"1f242f6c01da9a730308bae9a019d036530c0e496cec381a6e63214a9b10b1c8"} Oct 30 06:13:51.652607 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.652578 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:13:51.699956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.699924 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace1f0f2-b07b-4698-affb-482ce2bf4a00-kube-api-access-whhhg\" (UniqueName: \"kubernetes.io/projected/ace1f0f2-b07b-4698-affb-482ce2bf4a00-kube-api-access-whhhg\") pod \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\" (UID: \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\") " Oct 30 06:13:51.700123 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.700107 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace1f0f2-b07b-4698-affb-482ce2bf4a00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace1f0f2-b07b-4698-affb-482ce2bf4a00-plugins-config-volume\") pod \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\" (UID: \"ace1f0f2-b07b-4698-affb-482ce2bf4a00\") " Oct 30 06:13:51.700474 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.700446 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace1f0f2-b07b-4698-affb-482ce2bf4a00-plugins-config-volume" pod "ace1f0f2-b07b-4698-affb-482ce2bf4a00" (UID: "ace1f0f2-b07b-4698-affb-482ce2bf4a00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:51.700724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.700707 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ace1f0f2-b07b-4698-affb-482ce2bf4a00-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:13:51.702062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.702032 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace1f0f2-b07b-4698-affb-482ce2bf4a00-kube-api-access-whhhg" pod "ace1f0f2-b07b-4698-affb-482ce2bf4a00" (UID: "ace1f0f2-b07b-4698-affb-482ce2bf4a00"). InnerVolumeSpecName "kube-api-access-whhhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:51.801757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:51.801732 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whhhg\" (UniqueName: \"kubernetes.io/projected/ace1f0f2-b07b-4698-affb-482ce2bf4a00-kube-api-access-whhhg\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:13:52.524071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:52.524040 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" Oct 30 06:13:52.524576 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:52.524069 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9" event={"ID":"ace1f0f2-b07b-4698-affb-482ce2bf4a00","Type":"ContainerDied","Data":"c6e06b95c6bfcb200eb3fdcc549985bb2246d5e516547d9caa7f0ae044e0df90"} Oct 30 06:13:52.524576 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:52.524128 4431 scope.go:122] "RemoveContainer" containerID="1f242f6c01da9a730308bae9a019d036530c0e496cec381a6e63214a9b10b1c8" Oct 30 06:13:52.585437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:52.585413 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9"] Oct 30 06:13:52.594751 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:52.594730 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1ec766-745546f55f-9t8z9"] Oct 30 06:13:53.338739 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:13:53.338708 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace1f0f2-b07b-4698-affb-482ce2bf4a00" path="/var/lib/kubelet/pods/ace1f0f2-b07b-4698-affb-482ce2bf4a00/volumes" Oct 30 06:14:03.700117 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:14:03.700078 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:14:09.165913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.165874 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp"] Oct 30 06:14:09.166704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.166685 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ace1f0f2-b07b-4698-affb-482ce2bf4a00" containerName="payload-pre-processing" Oct 30 06:14:09.166704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.166705 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace1f0f2-b07b-4698-affb-482ce2bf4a00" containerName="payload-pre-processing" Oct 30 06:14:09.166811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.166805 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace1f0f2-b07b-4698-affb-482ce2bf4a00" containerName="payload-pre-processing" Oct 30 06:14:09.170379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.170356 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.195296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.195274 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e6df62-openshift-default-dockercfg-9d68h\"" Oct 30 06:14:09.212152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.212126 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp"] Oct 30 06:14:09.245437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.245401 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.245624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.245516 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-data\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.245624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.245551 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-istio-token\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.245624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.245611 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9skfb\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-kube-api-access-9skfb\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.245750 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.245667 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14733875-0c43-4209-935d-7df1b37133aa-istio-podinfo\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.245750 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.245689 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-credential-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.245750 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.245727 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14733875-0c43-4209-935d-7df1b37133aa-istiod-ca-cert\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.245860 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.245836 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-certs\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.245902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.245868 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-envoy\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.346777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.346743 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-certs\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.346777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.346778 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-envoy\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.346958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.346908 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.346958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.346949 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-data\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.346971 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-istio-token\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.346992 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skfb\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-kube-api-access-9skfb\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.347008 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14733875-0c43-4209-935d-7df1b37133aa-istio-podinfo\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.347034 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-credential-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.347087 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14733875-0c43-4209-935d-7df1b37133aa-istiod-ca-cert\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.347148 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-certs\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.347377 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-data\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.347514 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.347609 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-credential-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.347891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.347872 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14733875-0c43-4209-935d-7df1b37133aa-istiod-ca-cert\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.349177 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.349148 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-envoy\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.349584 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.349565 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14733875-0c43-4209-935d-7df1b37133aa-istio-podinfo\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.369868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.369839 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-istio-token\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.370053 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.370035 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skfb\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-kube-api-access-9skfb\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.483626 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.483603 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:09.809145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.809118 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp"] Oct 30 06:14:09.817858 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:14:09.817814 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14733875_0c43_4209_935d_7df1b37133aa.slice/crio-b42a205b853e00f196543a78ba994de35fa76d67b6fd5744052d4bb080b35f49 WatchSource:0}: Error finding container b42a205b853e00f196543a78ba994de35fa76d67b6fd5744052d4bb080b35f49: Status 404 returned error can't find the container with id b42a205b853e00f196543a78ba994de35fa76d67b6fd5744052d4bb080b35f49 Oct 30 06:14:09.820125 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.820088 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:14:09.820216 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.820183 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:14:09.820260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:09.820232 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:14:10.700178 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:10.700138 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" event={"ID":"14733875-0c43-4209-935d-7df1b37133aa","Type":"ContainerStarted","Data":"29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c"} Oct 30 06:14:10.700603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:10.700233 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" event={"ID":"14733875-0c43-4209-935d-7df1b37133aa","Type":"ContainerStarted","Data":"b42a205b853e00f196543a78ba994de35fa76d67b6fd5744052d4bb080b35f49"} Oct 30 06:14:10.783437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:10.770552 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" podStartSLOduration=1.7705324180000002 podStartE2EDuration="1.770532418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:10.75662054 +0000 UTC m=+1259.563142287" watchObservedRunningTime="2026-10-30 06:14:10.770532418 +0000 UTC m=+1259.577054167" Oct 30 06:14:11.485033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:11.484994 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:11.486611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:11.486583 4431 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 06:14:11.486752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:11.486633 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" podUID="14733875-0c43-4209-935d-7df1b37133aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 06:14:12.484692 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:12.484655 4431 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 06:14:12.485193 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:12.484790 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" podUID="14733875-0c43-4209-935d-7df1b37133aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 06:14:13.484521 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:13.484484 4431 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 06:14:13.484699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:13.484543 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" podUID="14733875-0c43-4209-935d-7df1b37133aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 06:14:14.488725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:14.488687 4431 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 06:14:14.489357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:14.489291 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" podUID="14733875-0c43-4209-935d-7df1b37133aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 06:14:15.484628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:15.484577 4431 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 06:14:15.484850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:15.484642 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" podUID="14733875-0c43-4209-935d-7df1b37133aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 06:14:16.488368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:16.488336 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:16.488755 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:16.488631 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:16.489274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:16.489252 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:14:25.887737 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:25.887656 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7"] Oct 30 06:14:25.893038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:25.893015 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:25.909590 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:25.909569 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c15d3a\"" Oct 30 06:14:25.921668 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:25.921641 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c15d3a-dockercfg-xhwp8\"" Oct 30 06:14:25.936814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:25.936767 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7"] Oct 30 06:14:26.021422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.021382 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpkq\" (UniqueName: \"kubernetes.io/projected/0f7ed02f-0ca1-4f44-866d-9ee740228c47-kube-api-access-kcpkq\") pod \"payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7\" (UID: \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\") " pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:26.021711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.021559 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7ed02f-0ca1-4f44-866d-9ee740228c47-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7\" (UID: \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\") " pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:26.123458 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.123417 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpkq\" (UniqueName: \"kubernetes.io/projected/0f7ed02f-0ca1-4f44-866d-9ee740228c47-kube-api-access-kcpkq\") pod \"payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7\" (UID: \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\") " pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:26.123673 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.123556 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7ed02f-0ca1-4f44-866d-9ee740228c47-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7\" (UID: \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\") " pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:26.124627 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.124600 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7ed02f-0ca1-4f44-866d-9ee740228c47-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7\" (UID: \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\") " pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:26.148231 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.148173 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpkq\" (UniqueName: \"kubernetes.io/projected/0f7ed02f-0ca1-4f44-866d-9ee740228c47-kube-api-access-kcpkq\") pod \"payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7\" (UID: \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\") " pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:26.214748 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.214718 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:26.530467 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:14:26.530422 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7ed02f_0ca1_4f44_866d_9ee740228c47.slice/crio-a5090026866e35be41827e2e18a65c7c00e8b5d41c83a1292fea8272bb19208f WatchSource:0}: Error finding container a5090026866e35be41827e2e18a65c7c00e8b5d41c83a1292fea8272bb19208f: Status 404 returned error can't find the container with id a5090026866e35be41827e2e18a65c7c00e8b5d41c83a1292fea8272bb19208f Oct 30 06:14:26.530467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.530447 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7"] Oct 30 06:14:26.798786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.798674 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" event={"ID":"0f7ed02f-0ca1-4f44-866d-9ee740228c47","Type":"ContainerStarted","Data":"fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620"} Oct 30 06:14:26.798786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.798763 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" event={"ID":"0f7ed02f-0ca1-4f44-866d-9ee740228c47","Type":"ContainerStarted","Data":"a5090026866e35be41827e2e18a65c7c00e8b5d41c83a1292fea8272bb19208f"} Oct 30 06:14:26.799010 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.798788 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:26.848026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:26.847964 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" podStartSLOduration=1.847946739 podStartE2EDuration="1.847946739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:26.846796991 +0000 UTC m=+1275.653318735" watchObservedRunningTime="2026-10-30 06:14:26.847946739 +0000 UTC m=+1275.654468488" Oct 30 06:14:33.700448 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:14:33.700407 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:14:37.810356 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:37.810329 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:14:49.845423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:49.845385 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2q89w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:14:49.845854 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:49.845442 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:14:49.873627 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:49.873594 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9jxv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:14:49.873868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:14:49.873838 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:15:00.222142 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.222066 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9"] Oct 30 06:15:00.227523 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.227503 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:00.250496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.250454 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9"] Oct 30 06:15:00.308273 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.308250 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:15:00.391442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.391408 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7420c-5295-41b5-95d3-4a96be42189a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9\" (UID: \"a0b7420c-5295-41b5-95d3-4a96be42189a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:00.391601 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.391561 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bcwh\" (UniqueName: \"kubernetes.io/projected/a0b7420c-5295-41b5-95d3-4a96be42189a-kube-api-access-2bcwh\") pod \"maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9\" (UID: \"a0b7420c-5295-41b5-95d3-4a96be42189a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:00.492555 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.492474 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bcwh\" (UniqueName: \"kubernetes.io/projected/a0b7420c-5295-41b5-95d3-4a96be42189a-kube-api-access-2bcwh\") pod \"maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9\" (UID: \"a0b7420c-5295-41b5-95d3-4a96be42189a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:00.492929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.492904 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7420c-5295-41b5-95d3-4a96be42189a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9\" (UID: \"a0b7420c-5295-41b5-95d3-4a96be42189a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:00.493510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.493493 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7420c-5295-41b5-95d3-4a96be42189a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9\" (UID: \"a0b7420c-5295-41b5-95d3-4a96be42189a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:00.518760 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.518729 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bcwh\" (UniqueName: \"kubernetes.io/projected/a0b7420c-5295-41b5-95d3-4a96be42189a-kube-api-access-2bcwh\") pod \"maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9\" (UID: \"a0b7420c-5295-41b5-95d3-4a96be42189a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:00.544522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.544498 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:00.852112 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.852082 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9"] Oct 30 06:15:00.852918 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:15:00.852892 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b7420c_5295_41b5_95d3_4a96be42189a.slice/crio-2ba04756a461fe29f3c73705f1151c45112bcd85a8f1377d17fc0f95a9c3d088 WatchSource:0}: Error finding container 2ba04756a461fe29f3c73705f1151c45112bcd85a8f1377d17fc0f95a9c3d088: Status 404 returned error can't find the container with id 2ba04756a461fe29f3c73705f1151c45112bcd85a8f1377d17fc0f95a9c3d088 Oct 30 06:15:00.855070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:00.855055 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:15:01.032448 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:01.032415 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" event={"ID":"a0b7420c-5295-41b5-95d3-4a96be42189a","Type":"ContainerStarted","Data":"2ba04756a461fe29f3c73705f1151c45112bcd85a8f1377d17fc0f95a9c3d088"} Oct 30 06:15:03.700358 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:15:03.700294 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:15:04.057790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:04.057748 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" event={"ID":"a0b7420c-5295-41b5-95d3-4a96be42189a","Type":"ContainerStarted","Data":"e09eb3897ed239224f2df6e1146c515501a83a3c9bb0a2029583f4e6d8f3093c"} Oct 30 06:15:04.101520 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:04.101478 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" podStartSLOduration=1.745986363 podStartE2EDuration="4.101465263s" totalImagesPullingTime="2.3554789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.099888918 +0000 UTC m=+1312.906410665" watchObservedRunningTime="2026-10-30 06:15:04.101465263 +0000 UTC m=+1312.907987010" Oct 30 06:15:05.064949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:05.064910 4431 generic.go:356] "Generic (PLEG): container finished" podID="a0b7420c-5295-41b5-95d3-4a96be42189a" containerID="e09eb3897ed239224f2df6e1146c515501a83a3c9bb0a2029583f4e6d8f3093c" exitCode=60 Oct 30 06:15:05.065449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:05.064965 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" event={"ID":"a0b7420c-5295-41b5-95d3-4a96be42189a","Type":"ContainerDied","Data":"e09eb3897ed239224f2df6e1146c515501a83a3c9bb0a2029583f4e6d8f3093c"} Oct 30 06:15:05.065449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:05.065282 4431 scope.go:122] "RemoveContainer" containerID="e09eb3897ed239224f2df6e1146c515501a83a3c9bb0a2029583f4e6d8f3093c" Oct 30 06:15:06.073829 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:06.073785 4431 generic.go:356] "Generic (PLEG): container finished" podID="a0b7420c-5295-41b5-95d3-4a96be42189a" containerID="af4262a9923803f4040a1eb803096dcc2a8f156648508e50a3939e942bff9685" exitCode=60 Oct 30 06:15:06.074257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:06.073834 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" event={"ID":"a0b7420c-5295-41b5-95d3-4a96be42189a","Type":"ContainerDied","Data":"af4262a9923803f4040a1eb803096dcc2a8f156648508e50a3939e942bff9685"} Oct 30 06:15:06.074257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:06.073889 4431 scope.go:122] "RemoveContainer" containerID="e09eb3897ed239224f2df6e1146c515501a83a3c9bb0a2029583f4e6d8f3093c" Oct 30 06:15:06.074257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:06.074161 4431 scope.go:122] "RemoveContainer" containerID="af4262a9923803f4040a1eb803096dcc2a8f156648508e50a3939e942bff9685" Oct 30 06:15:06.074495 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:15:06.074474 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9_odh-ai-gateway-infra(a0b7420c-5295-41b5-95d3-4a96be42189a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" podUID="a0b7420c-5295-41b5-95d3-4a96be42189a" Oct 30 06:15:07.082217 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:07.082191 4431 scope.go:122] "RemoveContainer" containerID="af4262a9923803f4040a1eb803096dcc2a8f156648508e50a3939e942bff9685" Oct 30 06:15:07.082642 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:15:07.082484 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9_odh-ai-gateway-infra(a0b7420c-5295-41b5-95d3-4a96be42189a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" podUID="a0b7420c-5295-41b5-95d3-4a96be42189a" Oct 30 06:15:10.057425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:10.057396 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9"] Oct 30 06:15:10.207467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:10.207430 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:10.334633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:10.334535 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b7420c-5295-41b5-95d3-4a96be42189a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7420c-5295-41b5-95d3-4a96be42189a-maas-api-service-ca\") pod \"a0b7420c-5295-41b5-95d3-4a96be42189a\" (UID: \"a0b7420c-5295-41b5-95d3-4a96be42189a\") " Oct 30 06:15:10.334815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:10.334653 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b7420c-5295-41b5-95d3-4a96be42189a-kube-api-access-2bcwh\" (UniqueName: \"kubernetes.io/projected/a0b7420c-5295-41b5-95d3-4a96be42189a-kube-api-access-2bcwh\") pod \"a0b7420c-5295-41b5-95d3-4a96be42189a\" (UID: \"a0b7420c-5295-41b5-95d3-4a96be42189a\") " Oct 30 06:15:10.335045 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:10.335003 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b7420c-5295-41b5-95d3-4a96be42189a-maas-api-service-ca" pod "a0b7420c-5295-41b5-95d3-4a96be42189a" (UID: "a0b7420c-5295-41b5-95d3-4a96be42189a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:10.335431 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:10.335406 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7420c-5295-41b5-95d3-4a96be42189a-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:10.336937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:10.336911 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b7420c-5295-41b5-95d3-4a96be42189a-kube-api-access-2bcwh" pod "a0b7420c-5295-41b5-95d3-4a96be42189a" (UID: "a0b7420c-5295-41b5-95d3-4a96be42189a"). InnerVolumeSpecName "kube-api-access-2bcwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:10.437386 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:10.437352 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bcwh\" (UniqueName: \"kubernetes.io/projected/a0b7420c-5295-41b5-95d3-4a96be42189a-kube-api-access-2bcwh\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:11.110263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:11.110224 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" Oct 30 06:15:11.110823 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:11.110226 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9" event={"ID":"a0b7420c-5295-41b5-95d3-4a96be42189a","Type":"ContainerDied","Data":"2ba04756a461fe29f3c73705f1151c45112bcd85a8f1377d17fc0f95a9c3d088"} Oct 30 06:15:11.110823 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:11.110365 4431 scope.go:122] "RemoveContainer" containerID="af4262a9923803f4040a1eb803096dcc2a8f156648508e50a3939e942bff9685" Oct 30 06:15:11.181096 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:11.181061 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9"] Oct 30 06:15:11.197828 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:11.197784 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f789d7-29889015-wb9s9"] Oct 30 06:15:11.339773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:11.339744 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b7420c-5295-41b5-95d3-4a96be42189a" path="/var/lib/kubelet/pods/a0b7420c-5295-41b5-95d3-4a96be42189a/volumes" Oct 30 06:15:33.701203 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:15:33.701161 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:15:46.884112 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:46.884077 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp"] Oct 30 06:15:46.884582 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:46.884339 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" podUID="14733875-0c43-4209-935d-7df1b37133aa" containerName="istio-proxy" containerID="cri-o://29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c" gracePeriod=30 Oct 30 06:15:52.144492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.144466 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:15:52.213569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.213533 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-istio-token\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-istio-token\") pod \"14733875-0c43-4209-935d-7df1b37133aa\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " Oct 30 06:15:52.213759 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.213601 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-credential-socket\") pod \"14733875-0c43-4209-935d-7df1b37133aa\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " Oct 30 06:15:52.213838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.213758 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-socket\") pod \"14733875-0c43-4209-935d-7df1b37133aa\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " Oct 30 06:15:52.213838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.213790 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14733875-0c43-4209-935d-7df1b37133aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14733875-0c43-4209-935d-7df1b37133aa-istiod-ca-cert\") pod \"14733875-0c43-4209-935d-7df1b37133aa\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " Oct 30 06:15:52.213947 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.213837 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14733875-0c43-4209-935d-7df1b37133aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14733875-0c43-4209-935d-7df1b37133aa-istio-podinfo\") pod \"14733875-0c43-4209-935d-7df1b37133aa\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " Oct 30 06:15:52.213947 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.213874 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-data\") pod \"14733875-0c43-4209-935d-7df1b37133aa\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " Oct 30 06:15:52.213947 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.213902 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-envoy\") pod \"14733875-0c43-4209-935d-7df1b37133aa\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " Oct 30 06:15:52.214111 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.214023 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-certs\") pod \"14733875-0c43-4209-935d-7df1b37133aa\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " Oct 30 06:15:52.214166 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.214100 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-socket" pod "14733875-0c43-4209-935d-7df1b37133aa" (UID: "14733875-0c43-4209-935d-7df1b37133aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.214222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.214165 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-kube-api-access-9skfb\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-kube-api-access-9skfb\") pod \"14733875-0c43-4209-935d-7df1b37133aa\" (UID: \"14733875-0c43-4209-935d-7df1b37133aa\") " Oct 30 06:15:52.214398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.214372 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-certs" pod "14733875-0c43-4209-935d-7df1b37133aa" (UID: "14733875-0c43-4209-935d-7df1b37133aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.214763 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.214625 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-data" pod "14733875-0c43-4209-935d-7df1b37133aa" (UID: "14733875-0c43-4209-935d-7df1b37133aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.215013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.214905 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.215013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.214929 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.215013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.214945 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.215013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.214979 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14733875-0c43-4209-935d-7df1b37133aa-istiod-ca-cert" pod "14733875-0c43-4209-935d-7df1b37133aa" (UID: "14733875-0c43-4209-935d-7df1b37133aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:52.215620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.215586 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-credential-socket" pod "14733875-0c43-4209-935d-7df1b37133aa" (UID: "14733875-0c43-4209-935d-7df1b37133aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.216690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.216660 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-istio-token" pod "14733875-0c43-4209-935d-7df1b37133aa" (UID: "14733875-0c43-4209-935d-7df1b37133aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:52.216964 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.216928 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-envoy" pod "14733875-0c43-4209-935d-7df1b37133aa" (UID: "14733875-0c43-4209-935d-7df1b37133aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.217031 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.216963 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14733875-0c43-4209-935d-7df1b37133aa-istio-podinfo" pod "14733875-0c43-4209-935d-7df1b37133aa" (UID: "14733875-0c43-4209-935d-7df1b37133aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:15:52.217260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.217241 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-kube-api-access-9skfb" pod "14733875-0c43-4209-935d-7df1b37133aa" (UID: "14733875-0c43-4209-935d-7df1b37133aa"). InnerVolumeSpecName "kube-api-access-9skfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:52.316441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.316412 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9skfb\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-kube-api-access-9skfb\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.316441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.316438 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14733875-0c43-4209-935d-7df1b37133aa-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.316441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.316448 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.316649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.316459 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14733875-0c43-4209-935d-7df1b37133aa-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.316649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.316468 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14733875-0c43-4209-935d-7df1b37133aa-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.316649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.316477 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14733875-0c43-4209-935d-7df1b37133aa-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.384296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.384268 4431 generic.go:356] "Generic (PLEG): container finished" podID="14733875-0c43-4209-935d-7df1b37133aa" containerID="29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c" exitCode=0 Oct 30 06:15:52.384452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.384335 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" Oct 30 06:15:52.384452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.384339 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" event={"ID":"14733875-0c43-4209-935d-7df1b37133aa","Type":"ContainerDied","Data":"29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c"} Oct 30 06:15:52.384452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.384404 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp" event={"ID":"14733875-0c43-4209-935d-7df1b37133aa","Type":"ContainerDied","Data":"b42a205b853e00f196543a78ba994de35fa76d67b6fd5744052d4bb080b35f49"} Oct 30 06:15:52.384452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.384435 4431 scope.go:122] "RemoveContainer" containerID="29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c" Oct 30 06:15:52.394605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.394585 4431 scope.go:122] "RemoveContainer" containerID="29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c" Oct 30 06:15:52.394881 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:15:52.394863 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c\": container with ID starting with 29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c not found: ID does not exist" containerID="29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c" Oct 30 06:15:52.394928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.394892 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c"} err="failed to get container status \"29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c\": rpc error: code = NotFound desc = could not find container \"29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c\": container with ID starting with 29f2f828e42dab398c2c56f41fbcb99399e0e946c064a33facedb906a7236a3c not found: ID does not exist" Oct 30 06:15:52.436658 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.436638 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp"] Oct 30 06:15:52.441787 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:52.441768 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-l4jkp"] Oct 30 06:15:53.343295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:15:53.343260 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14733875-0c43-4209-935d-7df1b37133aa" path="/var/lib/kubelet/pods/14733875-0c43-4209-935d-7df1b37133aa/volumes" Oct 30 06:16:02.493968 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:02.493936 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn"] Oct 30 06:16:03.701788 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:16:03.701756 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:16:07.741029 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:07.740996 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7"] Oct 30 06:16:12.742076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:12.742034 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" podUID="0f7ed02f-0ca1-4f44-866d-9ee740228c47" containerName="payload-processing" containerID="cri-o://fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620" gracePeriod=25 Oct 30 06:16:13.011079 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.011019 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:16:13.088977 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.088939 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f7ed02f-0ca1-4f44-866d-9ee740228c47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7ed02f-0ca1-4f44-866d-9ee740228c47-plugins-config-volume\") pod \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\" (UID: \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\") " Oct 30 06:16:13.089159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.089075 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7ed02f-0ca1-4f44-866d-9ee740228c47-kube-api-access-kcpkq\" (UniqueName: \"kubernetes.io/projected/0f7ed02f-0ca1-4f44-866d-9ee740228c47-kube-api-access-kcpkq\") pod \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\" (UID: \"0f7ed02f-0ca1-4f44-866d-9ee740228c47\") " Oct 30 06:16:13.089396 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.089370 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7ed02f-0ca1-4f44-866d-9ee740228c47-plugins-config-volume" pod "0f7ed02f-0ca1-4f44-866d-9ee740228c47" (UID: "0f7ed02f-0ca1-4f44-866d-9ee740228c47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:13.089551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.089535 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7ed02f-0ca1-4f44-866d-9ee740228c47-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:16:13.091476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.091445 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7ed02f-0ca1-4f44-866d-9ee740228c47-kube-api-access-kcpkq" pod "0f7ed02f-0ca1-4f44-866d-9ee740228c47" (UID: "0f7ed02f-0ca1-4f44-866d-9ee740228c47"). InnerVolumeSpecName "kube-api-access-kcpkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:13.190537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.190506 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcpkq\" (UniqueName: \"kubernetes.io/projected/0f7ed02f-0ca1-4f44-866d-9ee740228c47-kube-api-access-kcpkq\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:16:13.511423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.511393 4431 generic.go:356] "Generic (PLEG): container finished" podID="0f7ed02f-0ca1-4f44-866d-9ee740228c47" containerID="fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620" exitCode=0 Oct 30 06:16:13.511604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.511437 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" Oct 30 06:16:13.511604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.511489 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" event={"ID":"0f7ed02f-0ca1-4f44-866d-9ee740228c47","Type":"ContainerDied","Data":"fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620"} Oct 30 06:16:13.511604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.511541 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7" event={"ID":"0f7ed02f-0ca1-4f44-866d-9ee740228c47","Type":"ContainerDied","Data":"a5090026866e35be41827e2e18a65c7c00e8b5d41c83a1292fea8272bb19208f"} Oct 30 06:16:13.511604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.511572 4431 scope.go:122] "RemoveContainer" containerID="fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620" Oct 30 06:16:13.520708 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.520676 4431 scope.go:122] "RemoveContainer" containerID="fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620" Oct 30 06:16:13.520963 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:16:13.520946 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620\": container with ID starting with fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620 not found: ID does not exist" containerID="fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620" Oct 30 06:16:13.521009 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.520972 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620"} err="failed to get container status \"fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620\": rpc error: code = NotFound desc = could not find container \"fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620\": container with ID starting with fc22c3a8c5aaed3584ed15501f3289bc43a18a7f9974d342d4c5c1188869e620 not found: ID does not exist" Oct 30 06:16:13.566213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.566185 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7"] Oct 30 06:16:13.581443 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:13.581409 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c15d3a-67bdf8c648-smxp7"] Oct 30 06:16:15.338395 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:15.338365 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f7ed02f-0ca1-4f44-866d-9ee740228c47" path="/var/lib/kubelet/pods/0f7ed02f-0ca1-4f44-866d-9ee740228c47/volumes" Oct 30 06:16:17.514866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.514803 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" podUID="43d99277-ebc7-448a-b6ca-a001b71eaa42" containerName="main" containerID="cri-o://ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98" gracePeriod=45 Oct 30 06:16:17.803912 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.803889 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:16:17.951072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.951044 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d99277-ebc7-448a-b6ca-a001b71eaa42-tls-certs\" (UniqueName: \"kubernetes.io/secret/43d99277-ebc7-448a-b6ca-a001b71eaa42-tls-certs\") pod \"43d99277-ebc7-448a-b6ca-a001b71eaa42\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " Oct 30 06:16:17.951240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.951231 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-model-cache\") pod \"43d99277-ebc7-448a-b6ca-a001b71eaa42\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " Oct 30 06:16:17.951343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.951330 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-dshm\") pod \"43d99277-ebc7-448a-b6ca-a001b71eaa42\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " Oct 30 06:16:17.951411 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.951366 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-tmp-dir\") pod \"43d99277-ebc7-448a-b6ca-a001b71eaa42\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " Oct 30 06:16:17.951469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.951429 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d99277-ebc7-448a-b6ca-a001b71eaa42-kube-api-access-x7f68\" (UniqueName: \"kubernetes.io/projected/43d99277-ebc7-448a-b6ca-a001b71eaa42-kube-api-access-x7f68\") pod \"43d99277-ebc7-448a-b6ca-a001b71eaa42\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " Oct 30 06:16:17.951469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.951450 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-model-cache" pod "43d99277-ebc7-448a-b6ca-a001b71eaa42" (UID: "43d99277-ebc7-448a-b6ca-a001b71eaa42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:17.951583 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.951507 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-home\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-home\") pod \"43d99277-ebc7-448a-b6ca-a001b71eaa42\" (UID: \"43d99277-ebc7-448a-b6ca-a001b71eaa42\") " Oct 30 06:16:17.951767 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.951693 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-tmp-dir" pod "43d99277-ebc7-448a-b6ca-a001b71eaa42" (UID: "43d99277-ebc7-448a-b6ca-a001b71eaa42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:17.951872 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.951792 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-home" pod "43d99277-ebc7-448a-b6ca-a001b71eaa42" (UID: "43d99277-ebc7-448a-b6ca-a001b71eaa42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:17.952133 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.952112 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:16:17.952252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.952137 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:16:17.952252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.952150 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:16:17.954119 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.954082 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-dshm" pod "43d99277-ebc7-448a-b6ca-a001b71eaa42" (UID: "43d99277-ebc7-448a-b6ca-a001b71eaa42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:17.954354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.954327 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d99277-ebc7-448a-b6ca-a001b71eaa42-tls-certs" pod "43d99277-ebc7-448a-b6ca-a001b71eaa42" (UID: "43d99277-ebc7-448a-b6ca-a001b71eaa42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:17.954488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:17.954383 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d99277-ebc7-448a-b6ca-a001b71eaa42-kube-api-access-x7f68" pod "43d99277-ebc7-448a-b6ca-a001b71eaa42" (UID: "43d99277-ebc7-448a-b6ca-a001b71eaa42"). InnerVolumeSpecName "kube-api-access-x7f68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:18.053459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.053390 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d99277-ebc7-448a-b6ca-a001b71eaa42-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:16:18.053459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.053414 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7f68\" (UniqueName: \"kubernetes.io/projected/43d99277-ebc7-448a-b6ca-a001b71eaa42-kube-api-access-x7f68\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:16:18.053459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.053424 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d99277-ebc7-448a-b6ca-a001b71eaa42-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:16:18.544068 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.544034 4431 generic.go:356] "Generic (PLEG): container finished" podID="43d99277-ebc7-448a-b6ca-a001b71eaa42" containerID="ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98" exitCode=0 Oct 30 06:16:18.544530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.544078 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" Oct 30 06:16:18.544530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.544077 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" event={"ID":"43d99277-ebc7-448a-b6ca-a001b71eaa42","Type":"ContainerDied","Data":"ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98"} Oct 30 06:16:18.544530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.544180 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn" event={"ID":"43d99277-ebc7-448a-b6ca-a001b71eaa42","Type":"ContainerDied","Data":"e5fc4b5f150062deb195428a9d92d750ebbdfabc4470eba68eee76411797d43a"} Oct 30 06:16:18.544530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.544198 4431 scope.go:122] "RemoveContainer" containerID="ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98" Oct 30 06:16:18.555467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.555452 4431 scope.go:122] "RemoveContainer" containerID="ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98" Oct 30 06:16:18.555756 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:16:18.555737 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98\": container with ID starting with ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98 not found: ID does not exist" containerID="ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98" Oct 30 06:16:18.555914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.555764 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98"} err="failed to get container status \"ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98\": rpc error: code = NotFound desc = could not find container \"ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98\": container with ID starting with ae12fd199281871d87f6cb4124f832b2098750910c262708a063871d8d013e98 not found: ID does not exist" Oct 30 06:16:18.595781 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.595757 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn"] Oct 30 06:16:18.607692 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:18.607668 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ba44b248/e2e-lifecycle-model-ba44b248-kserve-d7844cc77-sd2zn"] Oct 30 06:16:19.338542 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:16:19.338510 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d99277-ebc7-448a-b6ca-a001b71eaa42" path="/var/lib/kubelet/pods/43d99277-ebc7-448a-b6ca-a001b71eaa42/volumes" Oct 30 06:16:33.705964 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:16:33.705932 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:17:03.714770 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:17:03.714732 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:17:15.207882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.207847 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm"] Oct 30 06:17:15.208425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208411 4431 cpu_manager.go:414] "containerMap: removing container" podUID="43d99277-ebc7-448a-b6ca-a001b71eaa42" containerName="main" Oct 30 06:17:15.208473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208426 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d99277-ebc7-448a-b6ca-a001b71eaa42" containerName="main" Oct 30 06:17:15.208473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208452 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a0b7420c-5295-41b5-95d3-4a96be42189a" containerName="cleanup" Oct 30 06:17:15.208473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208459 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b7420c-5295-41b5-95d3-4a96be42189a" containerName="cleanup" Oct 30 06:17:15.208473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208470 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a0b7420c-5295-41b5-95d3-4a96be42189a" containerName="cleanup" Oct 30 06:17:15.208597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208475 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b7420c-5295-41b5-95d3-4a96be42189a" containerName="cleanup" Oct 30 06:17:15.208597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208495 4431 cpu_manager.go:414] "containerMap: removing container" podUID="0f7ed02f-0ca1-4f44-866d-9ee740228c47" containerName="payload-processing" Oct 30 06:17:15.208597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208500 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7ed02f-0ca1-4f44-866d-9ee740228c47" containerName="payload-processing" Oct 30 06:17:15.208597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208510 4431 cpu_manager.go:414] "containerMap: removing container" podUID="14733875-0c43-4209-935d-7df1b37133aa" containerName="istio-proxy" Oct 30 06:17:15.208597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208516 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14733875-0c43-4209-935d-7df1b37133aa" containerName="istio-proxy" Oct 30 06:17:15.208745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208610 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d99277-ebc7-448a-b6ca-a001b71eaa42" containerName="main" Oct 30 06:17:15.208745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208620 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b7420c-5295-41b5-95d3-4a96be42189a" containerName="cleanup" Oct 30 06:17:15.208745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208626 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b7420c-5295-41b5-95d3-4a96be42189a" containerName="cleanup" Oct 30 06:17:15.208745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208635 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7ed02f-0ca1-4f44-866d-9ee740228c47" containerName="payload-processing" Oct 30 06:17:15.208745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.208641 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="14733875-0c43-4209-935d-7df1b37133aa" containerName="istio-proxy" Oct 30 06:17:15.213329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.213291 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:15.231726 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.231703 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm"] Oct 30 06:17:15.240266 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.240244 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f50f6e\"" Oct 30 06:17:15.240417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.240329 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f50f6e-dockercfg-t79tl\"" Oct 30 06:17:15.308654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.308602 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df469514-799e-414b-ac44-ff568371725d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm\" (UID: \"df469514-799e-414b-ac44-ff568371725d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:15.308848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.308796 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkkq6\" (UniqueName: \"kubernetes.io/projected/df469514-799e-414b-ac44-ff568371725d-kube-api-access-bkkq6\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm\" (UID: \"df469514-799e-414b-ac44-ff568371725d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:15.409694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.409658 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkkq6\" (UniqueName: \"kubernetes.io/projected/df469514-799e-414b-ac44-ff568371725d-kube-api-access-bkkq6\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm\" (UID: \"df469514-799e-414b-ac44-ff568371725d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:15.409861 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.409803 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df469514-799e-414b-ac44-ff568371725d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm\" (UID: \"df469514-799e-414b-ac44-ff568371725d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:15.410527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.410507 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df469514-799e-414b-ac44-ff568371725d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm\" (UID: \"df469514-799e-414b-ac44-ff568371725d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:15.433074 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.433044 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkkq6\" (UniqueName: \"kubernetes.io/projected/df469514-799e-414b-ac44-ff568371725d-kube-api-access-bkkq6\") pod \"payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm\" (UID: \"df469514-799e-414b-ac44-ff568371725d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:15.526400 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.526293 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:15.838742 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.838713 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm"] Oct 30 06:17:15.839750 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:17:15.839724 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf469514_799e_414b_ac44_ff568371725d.slice/crio-32f96dc8d7d4f1747a855bac8c903933cff50c5d35534fdf2a22c1b36962257f WatchSource:0}: Error finding container 32f96dc8d7d4f1747a855bac8c903933cff50c5d35534fdf2a22c1b36962257f: Status 404 returned error can't find the container with id 32f96dc8d7d4f1747a855bac8c903933cff50c5d35534fdf2a22c1b36962257f Oct 30 06:17:15.988493 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.988453 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" event={"ID":"df469514-799e-414b-ac44-ff568371725d","Type":"ContainerStarted","Data":"f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13"} Oct 30 06:17:15.988625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.988524 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" event={"ID":"df469514-799e-414b-ac44-ff568371725d","Type":"ContainerStarted","Data":"32f96dc8d7d4f1747a855bac8c903933cff50c5d35534fdf2a22c1b36962257f"} Oct 30 06:17:15.988730 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:15.988702 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:16.035700 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:16.035650 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" podStartSLOduration=1.035634184 podStartE2EDuration="1.035634184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:16.032777448 +0000 UTC m=+1444.839299196" watchObservedRunningTime="2026-10-30 06:17:16.035634184 +0000 UTC m=+1444.842155938" Oct 30 06:17:26.998645 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:26.998611 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:17:33.702063 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:17:33.702017 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:17:36.081632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:36.081599 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt"] Oct 30 06:17:36.082189 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:36.081817 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" containerID="cri-o://98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9" gracePeriod=30 Oct 30 06:17:41.383859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.383831 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:17:41.472779 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.472737 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-certs\") pod \"5650c0a7-778e-4156-af63-0ddd3c37667c\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " Oct 30 06:17:41.472965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.472795 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5650c0a7-778e-4156-af63-0ddd3c37667c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5650c0a7-778e-4156-af63-0ddd3c37667c-istiod-ca-cert\") pod \"5650c0a7-778e-4156-af63-0ddd3c37667c\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " Oct 30 06:17:41.472965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.472846 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-podinfo\") pod \"5650c0a7-778e-4156-af63-0ddd3c37667c\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " Oct 30 06:17:41.472965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.472894 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-kube-api-access-b4kvs\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-kube-api-access-b4kvs\") pod \"5650c0a7-778e-4156-af63-0ddd3c37667c\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " Oct 30 06:17:41.472965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.472928 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-token\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-token\") pod \"5650c0a7-778e-4156-af63-0ddd3c37667c\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " Oct 30 06:17:41.479835 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.479799 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5650c0a7-778e-4156-af63-0ddd3c37667c-istiod-ca-cert" pod "5650c0a7-778e-4156-af63-0ddd3c37667c" (UID: "5650c0a7-778e-4156-af63-0ddd3c37667c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:41.480355 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.480327 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-certs" pod "5650c0a7-778e-4156-af63-0ddd3c37667c" (UID: "5650c0a7-778e-4156-af63-0ddd3c37667c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.481466 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.481445 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-envoy\") pod \"5650c0a7-778e-4156-af63-0ddd3c37667c\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " Oct 30 06:17:41.481644 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.481625 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-credential-socket\") pod \"5650c0a7-778e-4156-af63-0ddd3c37667c\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " Oct 30 06:17:41.481860 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.481840 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-socket\") pod \"5650c0a7-778e-4156-af63-0ddd3c37667c\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " Oct 30 06:17:41.482461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.482164 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-socket" pod "5650c0a7-778e-4156-af63-0ddd3c37667c" (UID: "5650c0a7-778e-4156-af63-0ddd3c37667c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.482461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.482207 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-data\") pod \"5650c0a7-778e-4156-af63-0ddd3c37667c\" (UID: \"5650c0a7-778e-4156-af63-0ddd3c37667c\") " Oct 30 06:17:41.482461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.482372 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-credential-socket" pod "5650c0a7-778e-4156-af63-0ddd3c37667c" (UID: "5650c0a7-778e-4156-af63-0ddd3c37667c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.482461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.482428 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-data" pod "5650c0a7-778e-4156-af63-0ddd3c37667c" (UID: "5650c0a7-778e-4156-af63-0ddd3c37667c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.482910 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.482882 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.482998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.482919 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.482998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.482934 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5650c0a7-778e-4156-af63-0ddd3c37667c-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.482998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.482948 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.482998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.482963 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.484740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.484712 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-podinfo" pod "5650c0a7-778e-4156-af63-0ddd3c37667c" (UID: "5650c0a7-778e-4156-af63-0ddd3c37667c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:41.496647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.496592 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-token" pod "5650c0a7-778e-4156-af63-0ddd3c37667c" (UID: "5650c0a7-778e-4156-af63-0ddd3c37667c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.501411 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.501338 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-envoy" pod "5650c0a7-778e-4156-af63-0ddd3c37667c" (UID: "5650c0a7-778e-4156-af63-0ddd3c37667c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.501582 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.501555 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-kube-api-access-b4kvs" pod "5650c0a7-778e-4156-af63-0ddd3c37667c" (UID: "5650c0a7-778e-4156-af63-0ddd3c37667c"). InnerVolumeSpecName "kube-api-access-b4kvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.584560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.584526 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.584560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.584557 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.584814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.584568 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4kvs\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-kube-api-access-b4kvs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.584814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:41.584584 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5650c0a7-778e-4156-af63-0ddd3c37667c-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:42.204886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:42.204851 4431 generic.go:356] "Generic (PLEG): container finished" podID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerID="98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9" exitCode=0 Oct 30 06:17:42.205058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:42.204903 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" Oct 30 06:17:42.205058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:42.204928 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" event={"ID":"5650c0a7-778e-4156-af63-0ddd3c37667c","Type":"ContainerDied","Data":"98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9"} Oct 30 06:17:42.205058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:42.204980 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt" event={"ID":"5650c0a7-778e-4156-af63-0ddd3c37667c","Type":"ContainerDied","Data":"6e0f99418c14889c8da6d8ea3491362416c5b65bae1dabdc344a4b2747ff5567"} Oct 30 06:17:42.205058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:42.205002 4431 scope.go:122] "RemoveContainer" containerID="98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9" Oct 30 06:17:42.215448 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:42.215430 4431 scope.go:122] "RemoveContainer" containerID="98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9" Oct 30 06:17:42.215766 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:17:42.215743 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9\": container with ID starting with 98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9 not found: ID does not exist" containerID="98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9" Oct 30 06:17:42.215843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:42.215775 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9"} err="failed to get container status \"98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9\": rpc error: code = NotFound desc = could not find container \"98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9\": container with ID starting with 98a40cf15f64456900a84c9fe242ba39fd46b83e5d75cbc4596a8ce89b841ec9 not found: ID does not exist" Oct 30 06:17:42.275108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:42.275064 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt"] Oct 30 06:17:42.284247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:42.284218 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-9jhqt"] Oct 30 06:17:43.348467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:43.348426 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" path="/var/lib/kubelet/pods/5650c0a7-778e-4156-af63-0ddd3c37667c/volumes" Oct 30 06:17:49.581990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.581952 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr"] Oct 30 06:17:49.582581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.582565 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" Oct 30 06:17:49.582633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.582583 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" Oct 30 06:17:49.582716 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.582705 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5650c0a7-778e-4156-af63-0ddd3c37667c" containerName="istio-proxy" Oct 30 06:17:49.587146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.587124 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.625375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.625350 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr"] Oct 30 06:17:49.633134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.633100 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0fab6d-openshift-default-dockercfg-7pz45\"" Oct 30 06:17:49.695961 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.695930 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.696116 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.695978 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.696116 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.696002 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.696116 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.696068 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.696242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.696216 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8117705-4f64-42fe-93e5-1425a3e1087f-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.696280 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.696252 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqjq\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-kube-api-access-jvqjq\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.696405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.696384 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.696455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.696441 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.696569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.696553 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798127 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798170 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798230 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798297 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798366 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798392 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798409 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798444 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8117705-4f64-42fe-93e5-1425a3e1087f-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798460 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqjq\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-kube-api-access-jvqjq\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798608 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798683 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798758 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.798882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.798794 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.799221 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.799202 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8117705-4f64-42fe-93e5-1425a3e1087f-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.800511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.800492 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.800775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.800757 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.825685 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.825663 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.826956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.826935 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqjq\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-kube-api-access-jvqjq\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:49.900493 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:49.900443 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:50.221913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:50.221888 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr"] Oct 30 06:17:50.223220 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:17:50.223195 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8117705_4f64_42fe_93e5_1425a3e1087f.slice/crio-476dd96596b942a04c6f13f69335f0bcdcf3e0bc4d4af8ad0386e84d024d3f3c WatchSource:0}: Error finding container 476dd96596b942a04c6f13f69335f0bcdcf3e0bc4d4af8ad0386e84d024d3f3c: Status 404 returned error can't find the container with id 476dd96596b942a04c6f13f69335f0bcdcf3e0bc4d4af8ad0386e84d024d3f3c Oct 30 06:17:50.225582 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:50.225550 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:17:50.225811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:50.225616 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:17:50.225811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:50.225649 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:17:50.258210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:50.258183 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" event={"ID":"e8117705-4f64-42fe-93e5-1425a3e1087f","Type":"ContainerStarted","Data":"476dd96596b942a04c6f13f69335f0bcdcf3e0bc4d4af8ad0386e84d024d3f3c"} Oct 30 06:17:51.297168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:51.297115 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" event={"ID":"e8117705-4f64-42fe-93e5-1425a3e1087f","Type":"ContainerStarted","Data":"e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33"} Oct 30 06:17:51.431534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:51.431475 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" podStartSLOduration=2.431456592 podStartE2EDuration="2.431456592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:51.422252012 +0000 UTC m=+1480.228773760" watchObservedRunningTime="2026-10-30 06:17:51.431456592 +0000 UTC m=+1480.237978341" Oct 30 06:17:51.901602 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:51.901564 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:51.903515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:51.903479 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 06:17:51.903633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:51.903532 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 06:17:52.901786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:52.901753 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 06:17:52.902246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:52.901810 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 06:17:52.997219 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:52.997184 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz"] Oct 30 06:17:52.998037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:52.998001 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" containerID="cri-o://1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c" gracePeriod=30 Oct 30 06:17:53.003735 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:53.003709 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp"] Oct 30 06:17:53.003944 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:53.003918 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" containerID="cri-o://a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a" gracePeriod=30 Oct 30 06:17:53.008497 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:53.008474 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f"] Oct 30 06:17:53.008937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:53.008914 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" containerID="cri-o://663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54" gracePeriod=30 Oct 30 06:17:53.901385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:53.901270 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 06:17:53.901385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:53.901364 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 06:17:54.901119 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:54.901087 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 06:17:54.901535 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:54.901135 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 06:17:55.903789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:55.903753 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 06:17:55.904252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:55.904225 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 06:17:56.905462 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:56.905431 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:56.905939 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:56.905922 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:56.906648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:56.906619 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:17:58.296094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.296027 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:17:58.325577 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.325550 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:17:58.348174 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.348154 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:17:58.350725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.350577 4431 generic.go:356] "Generic (PLEG): container finished" podID="57acb25d-0b12-411c-831e-a7006448e7f5" containerID="1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c" exitCode=0 Oct 30 06:17:58.350725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.350626 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" Oct 30 06:17:58.350725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.350656 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" event={"ID":"57acb25d-0b12-411c-831e-a7006448e7f5","Type":"ContainerDied","Data":"1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c"} Oct 30 06:17:58.350943 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.350733 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz" event={"ID":"57acb25d-0b12-411c-831e-a7006448e7f5","Type":"ContainerDied","Data":"26f5530cfe1fbeedd0f47fd3097b1905969d703dec90b73f8360a70d34dc86dc"} Oct 30 06:17:58.350943 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.350759 4431 scope.go:122] "RemoveContainer" containerID="1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c" Oct 30 06:17:58.361231 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.359695 4431 generic.go:356] "Generic (PLEG): container finished" podID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerID="663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54" exitCode=0 Oct 30 06:17:58.361231 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.360030 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" Oct 30 06:17:58.361231 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.360065 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" event={"ID":"ac68ecd6-eac6-4c11-94ec-495b86717e56","Type":"ContainerDied","Data":"663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54"} Oct 30 06:17:58.362443 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.362403 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f" event={"ID":"ac68ecd6-eac6-4c11-94ec-495b86717e56","Type":"ContainerDied","Data":"526096733bbeb53512cbf8a8600eac5dc9d21a9003f6d5cc480b1326f8cbad03"} Oct 30 06:17:58.364436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.364408 4431 generic.go:356] "Generic (PLEG): container finished" podID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerID="a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a" exitCode=0 Oct 30 06:17:58.364553 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.364459 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" Oct 30 06:17:58.364608 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.364566 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" event={"ID":"b12e6cc5-e97d-4476-ba06-ab9d011c57b5","Type":"ContainerDied","Data":"a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a"} Oct 30 06:17:58.364670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.364610 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp" event={"ID":"b12e6cc5-e97d-4476-ba06-ab9d011c57b5","Type":"ContainerDied","Data":"711aa2000b4bea9543ee935f3a9bc3c07cc6ed9d29302ddc63d46280d2c07ca5"} Oct 30 06:17:58.371863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.371830 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-certs\") pod \"57acb25d-0b12-411c-831e-a7006448e7f5\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " Oct 30 06:17:58.371950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.371885 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57acb25d-0b12-411c-831e-a7006448e7f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57acb25d-0b12-411c-831e-a7006448e7f5-istio-podinfo\") pod \"57acb25d-0b12-411c-831e-a7006448e7f5\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " Oct 30 06:17:58.371950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.371927 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-istio-token\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-istio-token\") pod \"57acb25d-0b12-411c-831e-a7006448e7f5\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " Oct 30 06:17:58.372031 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.371958 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-credential-socket\") pod \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " Oct 30 06:17:58.372031 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372002 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-data\") pod \"57acb25d-0b12-411c-831e-a7006448e7f5\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " Oct 30 06:17:58.372152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372042 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-kube-api-access-kg5ls\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-kube-api-access-kg5ls\") pod \"57acb25d-0b12-411c-831e-a7006448e7f5\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " Oct 30 06:17:58.372152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372098 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-credential-socket\") pod \"57acb25d-0b12-411c-831e-a7006448e7f5\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " Oct 30 06:17:58.372152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372127 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-socket\") pod \"57acb25d-0b12-411c-831e-a7006448e7f5\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " Oct 30 06:17:58.372244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372221 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istiod-ca-cert\") pod \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " Oct 30 06:17:58.372277 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372256 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-certs\") pod \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " Oct 30 06:17:58.372388 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372370 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-envoy\") pod \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " Oct 30 06:17:58.372436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372413 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-data\") pod \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " Oct 30 06:17:58.372469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372450 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-socket\") pod \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " Oct 30 06:17:58.372539 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372523 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57acb25d-0b12-411c-831e-a7006448e7f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57acb25d-0b12-411c-831e-a7006448e7f5-istiod-ca-cert\") pod \"57acb25d-0b12-411c-831e-a7006448e7f5\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " Oct 30 06:17:58.372609 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372594 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-podinfo\") pod \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " Oct 30 06:17:58.372655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372642 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-token\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-token\") pod \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " Oct 30 06:17:58.372705 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372690 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-kube-api-access-xmj5n\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-kube-api-access-xmj5n\") pod \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\" (UID: \"b12e6cc5-e97d-4476-ba06-ab9d011c57b5\") " Oct 30 06:17:58.372739 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.372728 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-envoy\") pod \"57acb25d-0b12-411c-831e-a7006448e7f5\" (UID: \"57acb25d-0b12-411c-831e-a7006448e7f5\") " Oct 30 06:17:58.373332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.373284 4431 scope.go:122] "RemoveContainer" containerID="1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c" Oct 30 06:17:58.373588 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.373564 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-certs" pod "b12e6cc5-e97d-4476-ba06-ab9d011c57b5" (UID: "b12e6cc5-e97d-4476-ba06-ab9d011c57b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.373794 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.373772 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-certs" pod "57acb25d-0b12-411c-831e-a7006448e7f5" (UID: "57acb25d-0b12-411c-831e-a7006448e7f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.374290 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:17:58.374168 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c\": container with ID starting with 1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c not found: ID does not exist" containerID="1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c" Oct 30 06:17:58.374290 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.374212 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c"} err="failed to get container status \"1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c\": rpc error: code = NotFound desc = could not find container \"1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c\": container with ID starting with 1ebe02caa0fd3f1e971d7d20d046e681a9e2cf879632a346bc9605df15c53f8c not found: ID does not exist" Oct 30 06:17:58.374290 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.374245 4431 scope.go:122] "RemoveContainer" containerID="663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54" Oct 30 06:17:58.374585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.374481 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-data" pod "b12e6cc5-e97d-4476-ba06-ab9d011c57b5" (UID: "b12e6cc5-e97d-4476-ba06-ab9d011c57b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.374703 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.374666 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-socket" pod "b12e6cc5-e97d-4476-ba06-ab9d011c57b5" (UID: "b12e6cc5-e97d-4476-ba06-ab9d011c57b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.375687 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.375662 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-socket" pod "57acb25d-0b12-411c-831e-a7006448e7f5" (UID: "57acb25d-0b12-411c-831e-a7006448e7f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.375919 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.375892 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-credential-socket" pod "57acb25d-0b12-411c-831e-a7006448e7f5" (UID: "57acb25d-0b12-411c-831e-a7006448e7f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.376026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.375921 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57acb25d-0b12-411c-831e-a7006448e7f5-istiod-ca-cert" pod "57acb25d-0b12-411c-831e-a7006448e7f5" (UID: "57acb25d-0b12-411c-831e-a7006448e7f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:58.376210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.376122 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-credential-socket" pod "b12e6cc5-e97d-4476-ba06-ab9d011c57b5" (UID: "b12e6cc5-e97d-4476-ba06-ab9d011c57b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.376286 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.376226 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-data" pod "57acb25d-0b12-411c-831e-a7006448e7f5" (UID: "57acb25d-0b12-411c-831e-a7006448e7f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.376489 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.376463 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istiod-ca-cert" pod "b12e6cc5-e97d-4476-ba06-ab9d011c57b5" (UID: "b12e6cc5-e97d-4476-ba06-ab9d011c57b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:58.380794 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.380710 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-envoy" pod "57acb25d-0b12-411c-831e-a7006448e7f5" (UID: "57acb25d-0b12-411c-831e-a7006448e7f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.380794 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.380727 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-kube-api-access-kg5ls" pod "57acb25d-0b12-411c-831e-a7006448e7f5" (UID: "57acb25d-0b12-411c-831e-a7006448e7f5"). InnerVolumeSpecName "kube-api-access-kg5ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:58.380794 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.380762 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-kube-api-access-xmj5n" pod "b12e6cc5-e97d-4476-ba06-ab9d011c57b5" (UID: "b12e6cc5-e97d-4476-ba06-ab9d011c57b5"). InnerVolumeSpecName "kube-api-access-xmj5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:58.380794 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.380762 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-envoy" pod "b12e6cc5-e97d-4476-ba06-ab9d011c57b5" (UID: "b12e6cc5-e97d-4476-ba06-ab9d011c57b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.380978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.380809 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-podinfo" pod "b12e6cc5-e97d-4476-ba06-ab9d011c57b5" (UID: "b12e6cc5-e97d-4476-ba06-ab9d011c57b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:58.380978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.380817 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-token" pod "b12e6cc5-e97d-4476-ba06-ab9d011c57b5" (UID: "b12e6cc5-e97d-4476-ba06-ab9d011c57b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:58.382230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.382210 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57acb25d-0b12-411c-831e-a7006448e7f5-istio-podinfo" pod "57acb25d-0b12-411c-831e-a7006448e7f5" (UID: "57acb25d-0b12-411c-831e-a7006448e7f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:58.382553 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.382528 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-istio-token" pod "57acb25d-0b12-411c-831e-a7006448e7f5" (UID: "57acb25d-0b12-411c-831e-a7006448e7f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:58.386700 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.386683 4431 scope.go:122] "RemoveContainer" containerID="663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54" Oct 30 06:17:58.386995 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:17:58.386973 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54\": container with ID starting with 663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54 not found: ID does not exist" containerID="663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54" Oct 30 06:17:58.387083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.387003 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54"} err="failed to get container status \"663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54\": rpc error: code = NotFound desc = could not find container \"663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54\": container with ID starting with 663ed463a4bf710fcd6b039e1b759055a1393556942d7c90a6d5ac4794c14e54 not found: ID does not exist" Oct 30 06:17:58.387083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.387027 4431 scope.go:122] "RemoveContainer" containerID="a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a" Oct 30 06:17:58.396357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.396296 4431 scope.go:122] "RemoveContainer" containerID="a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a" Oct 30 06:17:58.396645 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:17:58.396619 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a\": container with ID starting with a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a not found: ID does not exist" containerID="a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a" Oct 30 06:17:58.396736 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.396651 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a"} err="failed to get container status \"a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a\": rpc error: code = NotFound desc = could not find container \"a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a\": container with ID starting with a44cbf4b0fd8dd6a202d0c5ea775946f1c88a4e048aabad090619e040636bc2a not found: ID does not exist" Oct 30 06:17:58.474777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.474747 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac68ecd6-eac6-4c11-94ec-495b86717e56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac68ecd6-eac6-4c11-94ec-495b86717e56-istiod-ca-cert\") pod \"ac68ecd6-eac6-4c11-94ec-495b86717e56\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " Oct 30 06:17:58.474930 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.474851 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-credential-socket\") pod \"ac68ecd6-eac6-4c11-94ec-495b86717e56\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " Oct 30 06:17:58.475040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.474886 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-kube-api-access-9sdpv\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-kube-api-access-9sdpv\") pod \"ac68ecd6-eac6-4c11-94ec-495b86717e56\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " Oct 30 06:17:58.475772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475078 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-envoy\") pod \"ac68ecd6-eac6-4c11-94ec-495b86717e56\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " Oct 30 06:17:58.475772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475151 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-credential-socket" pod "ac68ecd6-eac6-4c11-94ec-495b86717e56" (UID: "ac68ecd6-eac6-4c11-94ec-495b86717e56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.475772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475160 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-certs\") pod \"ac68ecd6-eac6-4c11-94ec-495b86717e56\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " Oct 30 06:17:58.475772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475161 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac68ecd6-eac6-4c11-94ec-495b86717e56-istiod-ca-cert" pod "ac68ecd6-eac6-4c11-94ec-495b86717e56" (UID: "ac68ecd6-eac6-4c11-94ec-495b86717e56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:58.475772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475234 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-socket\") pod \"ac68ecd6-eac6-4c11-94ec-495b86717e56\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " Oct 30 06:17:58.475772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475279 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-podinfo\") pod \"ac68ecd6-eac6-4c11-94ec-495b86717e56\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " Oct 30 06:17:58.475772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475351 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-token\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-token\") pod \"ac68ecd6-eac6-4c11-94ec-495b86717e56\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " Oct 30 06:17:58.475772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475431 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-data\") pod \"ac68ecd6-eac6-4c11-94ec-495b86717e56\" (UID: \"ac68ecd6-eac6-4c11-94ec-495b86717e56\") " Oct 30 06:17:58.475772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475652 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-socket" pod "ac68ecd6-eac6-4c11-94ec-495b86717e56" (UID: "ac68ecd6-eac6-4c11-94ec-495b86717e56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.476394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.475831 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-certs" pod "ac68ecd6-eac6-4c11-94ec-495b86717e56" (UID: "ac68ecd6-eac6-4c11-94ec-495b86717e56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.476394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476276 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476313 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476328 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476343 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476357 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476369 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476382 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476397 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476412 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57acb25d-0b12-411c-831e-a7006448e7f5-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476428 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476441 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476455 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmj5n\" (UniqueName: \"kubernetes.io/projected/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-kube-api-access-xmj5n\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476469 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476481 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476493 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57acb25d-0b12-411c-831e-a7006448e7f5-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476508 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476527 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12e6cc5-e97d-4476-ba06-ab9d011c57b5-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476542 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476556 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5ls\" (UniqueName: \"kubernetes.io/projected/57acb25d-0b12-411c-831e-a7006448e7f5-kube-api-access-kg5ls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476568 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac68ecd6-eac6-4c11-94ec-495b86717e56-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476582 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.476822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.476595 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57acb25d-0b12-411c-831e-a7006448e7f5-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.478379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.478354 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-data" pod "ac68ecd6-eac6-4c11-94ec-495b86717e56" (UID: "ac68ecd6-eac6-4c11-94ec-495b86717e56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.478649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.478628 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-podinfo" pod "ac68ecd6-eac6-4c11-94ec-495b86717e56" (UID: "ac68ecd6-eac6-4c11-94ec-495b86717e56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:58.479508 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.479482 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-envoy" pod "ac68ecd6-eac6-4c11-94ec-495b86717e56" (UID: "ac68ecd6-eac6-4c11-94ec-495b86717e56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:58.480076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.480047 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-kube-api-access-9sdpv" pod "ac68ecd6-eac6-4c11-94ec-495b86717e56" (UID: "ac68ecd6-eac6-4c11-94ec-495b86717e56"). InnerVolumeSpecName "kube-api-access-9sdpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:58.480863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.480841 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-token" pod "ac68ecd6-eac6-4c11-94ec-495b86717e56" (UID: "ac68ecd6-eac6-4c11-94ec-495b86717e56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:58.578242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.578159 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.578242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.578197 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.578242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.578216 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sdpv\" (UniqueName: \"kubernetes.io/projected/ac68ecd6-eac6-4c11-94ec-495b86717e56-kube-api-access-9sdpv\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.578242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.578231 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.578242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.578245 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac68ecd6-eac6-4c11-94ec-495b86717e56-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:17:58.704190 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.704163 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz"] Oct 30 06:17:58.707618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.707598 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-v7xfz"] Oct 30 06:17:58.741329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.741292 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f"] Oct 30 06:17:58.745242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.745215 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-s7r9f"] Oct 30 06:17:58.778556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.778526 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp"] Oct 30 06:17:58.784365 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:58.784343 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-742qp"] Oct 30 06:17:59.291760 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:59.291663 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm"] Oct 30 06:17:59.341869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:59.341832 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" path="/var/lib/kubelet/pods/57acb25d-0b12-411c-831e-a7006448e7f5/volumes" Oct 30 06:17:59.342567 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:59.342544 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" path="/var/lib/kubelet/pods/ac68ecd6-eac6-4c11-94ec-495b86717e56/volumes" Oct 30 06:17:59.343041 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:17:59.343020 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" path="/var/lib/kubelet/pods/b12e6cc5-e97d-4476-ba06-ab9d011c57b5/volumes" Oct 30 06:18:03.704119 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:18:03.704083 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:18:04.293026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:04.292956 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" podUID="df469514-799e-414b-ac44-ff568371725d" containerName="payload-pre-processing" containerID="cri-o://f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13" gracePeriod=25 Oct 30 06:18:04.553805 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:04.553755 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:18:04.667347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:04.667295 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df469514-799e-414b-ac44-ff568371725d-kube-api-access-bkkq6\" (UniqueName: \"kubernetes.io/projected/df469514-799e-414b-ac44-ff568371725d-kube-api-access-bkkq6\") pod \"df469514-799e-414b-ac44-ff568371725d\" (UID: \"df469514-799e-414b-ac44-ff568371725d\") " Oct 30 06:18:04.667518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:04.667380 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df469514-799e-414b-ac44-ff568371725d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df469514-799e-414b-ac44-ff568371725d-plugins-config-volume\") pod \"df469514-799e-414b-ac44-ff568371725d\" (UID: \"df469514-799e-414b-ac44-ff568371725d\") " Oct 30 06:18:04.667760 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:04.667737 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df469514-799e-414b-ac44-ff568371725d-plugins-config-volume" pod "df469514-799e-414b-ac44-ff568371725d" (UID: "df469514-799e-414b-ac44-ff568371725d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:04.667976 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:04.667960 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df469514-799e-414b-ac44-ff568371725d-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.669394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:04.669369 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df469514-799e-414b-ac44-ff568371725d-kube-api-access-bkkq6" pod "df469514-799e-414b-ac44-ff568371725d" (UID: "df469514-799e-414b-ac44-ff568371725d"). InnerVolumeSpecName "kube-api-access-bkkq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:04.769518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:04.769490 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkkq6\" (UniqueName: \"kubernetes.io/projected/df469514-799e-414b-ac44-ff568371725d-kube-api-access-bkkq6\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:18:05.414046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:05.414014 4431 generic.go:356] "Generic (PLEG): container finished" podID="df469514-799e-414b-ac44-ff568371725d" containerID="f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13" exitCode=0 Oct 30 06:18:05.414180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:05.414056 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" event={"ID":"df469514-799e-414b-ac44-ff568371725d","Type":"ContainerDied","Data":"f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13"} Oct 30 06:18:05.414180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:05.414060 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" Oct 30 06:18:05.414180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:05.414097 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm" event={"ID":"df469514-799e-414b-ac44-ff568371725d","Type":"ContainerDied","Data":"32f96dc8d7d4f1747a855bac8c903933cff50c5d35534fdf2a22c1b36962257f"} Oct 30 06:18:05.414180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:05.414118 4431 scope.go:122] "RemoveContainer" containerID="f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13" Oct 30 06:18:05.423400 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:05.423382 4431 scope.go:122] "RemoveContainer" containerID="f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13" Oct 30 06:18:05.423686 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:18:05.423666 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13\": container with ID starting with f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13 not found: ID does not exist" containerID="f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13" Oct 30 06:18:05.423769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:05.423694 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13"} err="failed to get container status \"f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13\": rpc error: code = NotFound desc = could not find container \"f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13\": container with ID starting with f659f38e58991815e2404f95074381789a2a95e215ff431f29a250bd827c2b13 not found: ID does not exist" Oct 30 06:18:05.467118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:05.466999 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm"] Oct 30 06:18:05.477255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:05.477190 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f50f6e-7f47794f95-vx5bm"] Oct 30 06:18:07.337237 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:07.337203 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df469514-799e-414b-ac44-ff568371725d" path="/var/lib/kubelet/pods/df469514-799e-414b-ac44-ff568371725d/volumes" Oct 30 06:18:08.756850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.756818 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6"] Oct 30 06:18:08.757354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757341 4431 cpu_manager.go:414] "containerMap: removing container" podUID="df469514-799e-414b-ac44-ff568371725d" containerName="payload-pre-processing" Oct 30 06:18:08.757409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757355 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df469514-799e-414b-ac44-ff568371725d" containerName="payload-pre-processing" Oct 30 06:18:08.757409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757374 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" Oct 30 06:18:08.757409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757379 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" Oct 30 06:18:08.757409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757388 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" Oct 30 06:18:08.757409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757393 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" Oct 30 06:18:08.757564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757418 4431 cpu_manager.go:414] "containerMap: removing container" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" Oct 30 06:18:08.757564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757422 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" Oct 30 06:18:08.757564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757493 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="57acb25d-0b12-411c-831e-a7006448e7f5" containerName="istio-proxy" Oct 30 06:18:08.757564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757501 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="df469514-799e-414b-ac44-ff568371725d" containerName="payload-pre-processing" Oct 30 06:18:08.757564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757509 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac68ecd6-eac6-4c11-94ec-495b86717e56" containerName="istio-proxy" Oct 30 06:18:08.757564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.757515 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12e6cc5-e97d-4476-ba06-ab9d011c57b5" containerName="istio-proxy" Oct 30 06:18:08.761615 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.761598 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:08.783618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.783592 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6"] Oct 30 06:18:08.790889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.790860 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8272af-dockercfg-q7gzc\"" Oct 30 06:18:08.790991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.790892 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8272af\"" Oct 30 06:18:08.837923 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.837882 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6\" (UID: \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:08.838185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.838165 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5rf\" (UniqueName: \"kubernetes.io/projected/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-kube-api-access-hn5rf\") pod \"payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6\" (UID: \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:08.858826 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.858793 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk"] Oct 30 06:18:08.865165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.865133 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:08.885551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.885475 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk"] Oct 30 06:18:08.939614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.939577 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ea6618-0a2a-4142-99f2-83fdc378e095-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk\" (UID: \"b0ea6618-0a2a-4142-99f2-83fdc378e095\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:08.939779 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.939746 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5rf\" (UniqueName: \"kubernetes.io/projected/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-kube-api-access-hn5rf\") pod \"payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6\" (UID: \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:08.939838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.939824 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cds\" (UniqueName: \"kubernetes.io/projected/b0ea6618-0a2a-4142-99f2-83fdc378e095-kube-api-access-47cds\") pod \"payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk\" (UID: \"b0ea6618-0a2a-4142-99f2-83fdc378e095\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:08.939894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.939863 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6\" (UID: \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:08.940796 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.940773 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6\" (UID: \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:08.965829 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:08.965798 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5rf\" (UniqueName: \"kubernetes.io/projected/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-kube-api-access-hn5rf\") pod \"payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6\" (UID: \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:09.041376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:09.041261 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ea6618-0a2a-4142-99f2-83fdc378e095-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk\" (UID: \"b0ea6618-0a2a-4142-99f2-83fdc378e095\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:09.041546 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:09.041511 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47cds\" (UniqueName: \"kubernetes.io/projected/b0ea6618-0a2a-4142-99f2-83fdc378e095-kube-api-access-47cds\") pod \"payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk\" (UID: \"b0ea6618-0a2a-4142-99f2-83fdc378e095\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:09.041987 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:09.041966 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ea6618-0a2a-4142-99f2-83fdc378e095-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk\" (UID: \"b0ea6618-0a2a-4142-99f2-83fdc378e095\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:09.065472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:09.065438 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cds\" (UniqueName: \"kubernetes.io/projected/b0ea6618-0a2a-4142-99f2-83fdc378e095-kube-api-access-47cds\") pod \"payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk\" (UID: \"b0ea6618-0a2a-4142-99f2-83fdc378e095\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:09.075417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:09.075126 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:09.186787 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:09.186747 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:09.449838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:09.449814 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6"] Oct 30 06:18:09.451150 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:18:09.451122 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0171d1_c65f_4cef_a7a6_9d6e7b775bd0.slice/crio-5b82eeb294e0657d7028aa2d3e5b9aab89100ea6af85e9a8439b9e8aeb6052c9 WatchSource:0}: Error finding container 5b82eeb294e0657d7028aa2d3e5b9aab89100ea6af85e9a8439b9e8aeb6052c9: Status 404 returned error can't find the container with id 5b82eeb294e0657d7028aa2d3e5b9aab89100ea6af85e9a8439b9e8aeb6052c9 Oct 30 06:18:09.535339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:09.535315 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk"] Oct 30 06:18:09.536639 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:18:09.536618 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ea6618_0a2a_4142_99f2_83fdc378e095.slice/crio-cd49740f6f3bdbdac81e496e6e5217a5528e075656980b0fc09ed0b5b2a22cd5 WatchSource:0}: Error finding container cd49740f6f3bdbdac81e496e6e5217a5528e075656980b0fc09ed0b5b2a22cd5: Status 404 returned error can't find the container with id cd49740f6f3bdbdac81e496e6e5217a5528e075656980b0fc09ed0b5b2a22cd5 Oct 30 06:18:10.452252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:10.452209 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" event={"ID":"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0","Type":"ContainerStarted","Data":"ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510"} Oct 30 06:18:10.452702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:10.452280 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" event={"ID":"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0","Type":"ContainerStarted","Data":"5b82eeb294e0657d7028aa2d3e5b9aab89100ea6af85e9a8439b9e8aeb6052c9"} Oct 30 06:18:10.452702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:10.452334 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:10.453969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:10.453935 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" event={"ID":"b0ea6618-0a2a-4142-99f2-83fdc378e095","Type":"ContainerStarted","Data":"f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0"} Oct 30 06:18:10.454109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:10.453973 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" event={"ID":"b0ea6618-0a2a-4142-99f2-83fdc378e095","Type":"ContainerStarted","Data":"cd49740f6f3bdbdac81e496e6e5217a5528e075656980b0fc09ed0b5b2a22cd5"} Oct 30 06:18:10.454109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:10.454017 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:10.500998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:10.500944 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" podStartSLOduration=2.5009295700000003 podStartE2EDuration="2.50092957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:10.498359236 +0000 UTC m=+1499.304880985" watchObservedRunningTime="2026-10-30 06:18:10.50092957 +0000 UTC m=+1499.307451318" Oct 30 06:18:10.542464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:10.542398 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" podStartSLOduration=2.542377562 podStartE2EDuration="2.542377562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:10.540284381 +0000 UTC m=+1499.346806130" watchObservedRunningTime="2026-10-30 06:18:10.542377562 +0000 UTC m=+1499.348899316" Oct 30 06:18:12.137678 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.137642 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4"] Oct 30 06:18:12.142969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.142947 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.157896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.157868 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-532f3b-serving-cert\"" Oct 30 06:18:12.174698 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.174643 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4"] Oct 30 06:18:12.180593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.180563 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.180774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.180639 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7vnk\" (UniqueName: \"kubernetes.io/projected/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-kube-api-access-x7vnk\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.180774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.180663 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-metrics-certs\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.180774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.180736 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.281847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.281814 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.282008 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.281907 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7vnk\" (UniqueName: \"kubernetes.io/projected/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-kube-api-access-x7vnk\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.282008 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.281939 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-metrics-certs\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.282082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.282036 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.282609 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.282579 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.284501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.284476 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.284595 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.284502 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-metrics-certs\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.305370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.305349 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7vnk\" (UniqueName: \"kubernetes.io/projected/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-kube-api-access-x7vnk\") pod \"maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.456546 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.456503 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:12.760688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:12.760662 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4"] Oct 30 06:18:12.761941 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:18:12.761920 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1a329d_b0a9_459b_9dbb_8cb595aa640c.slice/crio-89a9d9e78dfe76dd6212ca25b851bf995d5b84f01ce408091dc43a16aa8e74f4 WatchSource:0}: Error finding container 89a9d9e78dfe76dd6212ca25b851bf995d5b84f01ce408091dc43a16aa8e74f4: Status 404 returned error can't find the container with id 89a9d9e78dfe76dd6212ca25b851bf995d5b84f01ce408091dc43a16aa8e74f4 Oct 30 06:18:13.477625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:13.477580 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" event={"ID":"ab1a329d-b0a9-459b-9dbb-8cb595aa640c","Type":"ContainerStarted","Data":"0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8"} Oct 30 06:18:13.481333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:13.477661 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" event={"ID":"ab1a329d-b0a9-459b-9dbb-8cb595aa640c","Type":"ContainerStarted","Data":"89a9d9e78dfe76dd6212ca25b851bf995d5b84f01ce408091dc43a16aa8e74f4"} Oct 30 06:18:13.481333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:13.478780 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:13.523337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:13.523271 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" podStartSLOduration=1.1252156150000001 podStartE2EDuration="1.52325599s" totalImagesPullingTime="398.040375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:13.521161399 +0000 UTC m=+1502.327683159" watchObservedRunningTime="2026-10-30 06:18:13.52325599 +0000 UTC m=+1502.329777738" Oct 30 06:18:20.497150 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:20.497114 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:18:21.463321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:21.463275 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:18:21.463501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:21.463491 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:18:28.206954 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.206920 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8"] Oct 30 06:18:28.216726 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.216698 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.231116 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.231089 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8"] Oct 30 06:18:28.238919 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.238886 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a55a4b-serving-cert\"" Oct 30 06:18:28.285339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.285295 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.285461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.285420 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-metrics-certs\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.285583 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.285568 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.285621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.285610 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgql\" (UniqueName: \"kubernetes.io/projected/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-kube-api-access-zbgql\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.386904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.386873 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.387047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.386925 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgql\" (UniqueName: \"kubernetes.io/projected/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-kube-api-access-zbgql\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.387047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.386988 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.387047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.387042 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-metrics-certs\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.387698 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.387673 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.389372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.389349 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.389495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.389475 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-metrics-certs\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.409560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.409530 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgql\" (UniqueName: \"kubernetes.io/projected/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-kube-api-access-zbgql\") pod \"maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.533497 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.533400 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:28.845258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:28.845228 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8"] Oct 30 06:18:28.845744 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:18:28.845724 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb45dcde_f629_4717_ad1c_66ad3bb86ac3.slice/crio-cd0d8c951c699309162311456f8d914d2eef6167c13f8e8eca0e10d2d1b33b7c WatchSource:0}: Error finding container cd0d8c951c699309162311456f8d914d2eef6167c13f8e8eca0e10d2d1b33b7c: Status 404 returned error can't find the container with id cd0d8c951c699309162311456f8d914d2eef6167c13f8e8eca0e10d2d1b33b7c Oct 30 06:18:29.578485 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:29.578441 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" event={"ID":"fb45dcde-f629-4717-ad1c-66ad3bb86ac3","Type":"ContainerStarted","Data":"2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33"} Oct 30 06:18:29.578890 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:29.578500 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" event={"ID":"fb45dcde-f629-4717-ad1c-66ad3bb86ac3","Type":"ContainerStarted","Data":"cd0d8c951c699309162311456f8d914d2eef6167c13f8e8eca0e10d2d1b33b7c"} Oct 30 06:18:29.578890 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:29.578526 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:29.620013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:29.619951 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" podStartSLOduration=1.340068577 podStartE2EDuration="1.619934329s" totalImagesPullingTime="279.865752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:29.619375842 +0000 UTC m=+1518.425897616" watchObservedRunningTime="2026-10-30 06:18:29.619934329 +0000 UTC m=+1518.426456078" Oct 30 06:18:33.701542 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:18:33.701470 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:18:34.720745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.720707 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq"] Oct 30 06:18:34.725678 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.725655 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.761054 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.761025 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq"] Oct 30 06:18:34.764296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.764268 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6"] Oct 30 06:18:34.769518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.769491 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.775169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.775140 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.775284 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.775201 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.775284 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.775217 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.775284 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.775261 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.775493 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.775296 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.775493 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.775416 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750fc01-847b-4584-8ec9-b039222088ee-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.775617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.775526 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750fc01-847b-4584-8ec9-b039222088ee-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.775617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.775561 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmqvs\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-kube-api-access-vmqvs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.775723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.775615 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.789946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.789917 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6"] Oct 30 06:18:34.876664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.876635 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqvs\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-kube-api-access-vmqvs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.876843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.876673 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.876843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.876744 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6964007e-22ee-4328-b525-7fc9cbb2ebea-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.876843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.876771 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.876843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.876798 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.876912 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.877064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.876951 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzt6n\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-kube-api-access-hzt6n\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.877064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877055 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.877235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877082 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877101 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877213 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877363 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877261 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877363 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877316 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877363 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877339 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.877486 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877387 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.877486 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877391 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877488 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750fc01-847b-4584-8ec9-b039222088ee-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877514 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.877557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877535 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877615 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.877709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877656 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.877709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.877700 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750fc01-847b-4584-8ec9-b039222088ee-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.878248 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.878228 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750fc01-847b-4584-8ec9-b039222088ee-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.879109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.879082 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.879813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.879792 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750fc01-847b-4584-8ec9-b039222088ee-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.899484 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.899461 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.899860 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.899835 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqvs\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-kube-api-access-vmqvs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:34.979222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979163 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6964007e-22ee-4328-b525-7fc9cbb2ebea-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979194 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979214 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979395 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzt6n\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-kube-api-access-hzt6n\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979464 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979546 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979522 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979606 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979554 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979606 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979597 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979615 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979659 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979725 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.979944 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979914 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6964007e-22ee-4328-b525-7fc9cbb2ebea-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.980058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.979976 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.980058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.980016 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.981495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.981472 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:34.981785 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:34.981768 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:35.002288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.002259 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:35.002393 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.002289 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzt6n\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-kube-api-access-hzt6n\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:35.043719 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.043701 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:35.089860 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.089833 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:35.364794 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.364764 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq"] Oct 30 06:18:35.365263 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:18:35.365237 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd750fc01_847b_4584_8ec9_b039222088ee.slice/crio-4014303935cd389bdcbc075ce2ba06b3f7e7f23cc75e2529cf80ef79cb85f8e0 WatchSource:0}: Error finding container 4014303935cd389bdcbc075ce2ba06b3f7e7f23cc75e2529cf80ef79cb85f8e0: Status 404 returned error can't find the container with id 4014303935cd389bdcbc075ce2ba06b3f7e7f23cc75e2529cf80ef79cb85f8e0 Oct 30 06:18:35.367823 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.367793 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:18:35.367884 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.367855 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:18:35.367925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.367884 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:18:35.410925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.410902 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6"] Oct 30 06:18:35.412472 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:18:35.412445 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6964007e_22ee_4328_b525_7fc9cbb2ebea.slice/crio-ec89e0d33090ad5a6f443365cfb13093b1cca138d22eafc215014ce283b58f7d WatchSource:0}: Error finding container ec89e0d33090ad5a6f443365cfb13093b1cca138d22eafc215014ce283b58f7d: Status 404 returned error can't find the container with id ec89e0d33090ad5a6f443365cfb13093b1cca138d22eafc215014ce283b58f7d Oct 30 06:18:35.414928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.414895 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:18:35.414998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.414962 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:18:35.414998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.414990 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:18:35.589275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.589247 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:18:35.623094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.623002 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" event={"ID":"d750fc01-847b-4584-8ec9-b039222088ee","Type":"ContainerStarted","Data":"be581c896c31a190e0a1ab0a51358d8467803595cb079316ac05b62cdfe59927"} Oct 30 06:18:35.623094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.623072 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" event={"ID":"d750fc01-847b-4584-8ec9-b039222088ee","Type":"ContainerStarted","Data":"4014303935cd389bdcbc075ce2ba06b3f7e7f23cc75e2529cf80ef79cb85f8e0"} Oct 30 06:18:35.625405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.625273 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" event={"ID":"6964007e-22ee-4328-b525-7fc9cbb2ebea","Type":"ContainerStarted","Data":"9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a"} Oct 30 06:18:35.625405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.625364 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" event={"ID":"6964007e-22ee-4328-b525-7fc9cbb2ebea","Type":"ContainerStarted","Data":"ec89e0d33090ad5a6f443365cfb13093b1cca138d22eafc215014ce283b58f7d"} Oct 30 06:18:35.678118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.678060 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" podStartSLOduration=1.678038676 podStartE2EDuration="1.678038676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:35.675576373 +0000 UTC m=+1524.482098109" watchObservedRunningTime="2026-10-30 06:18:35.678038676 +0000 UTC m=+1524.484560630" Oct 30 06:18:35.726772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:35.726723 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" podStartSLOduration=1.726705168 podStartE2EDuration="1.726705168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:35.723186581 +0000 UTC m=+1524.529708329" watchObservedRunningTime="2026-10-30 06:18:35.726705168 +0000 UTC m=+1524.533226920" Oct 30 06:18:36.044137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:36.044100 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:36.045667 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:36.045640 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:18:36.045811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:36.045686 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 06:18:36.093969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:36.093666 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:36.099324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:36.095160 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 06:18:36.099324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:36.095218 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 06:18:37.047782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:37.046093 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:18:37.047782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:37.046152 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 06:18:37.107335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:37.105403 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 06:18:37.107335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:37.105468 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 06:18:38.044779 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:38.044743 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:18:38.044967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:38.044795 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 06:18:38.093324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:38.092673 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 06:18:38.093324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:38.092729 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 06:18:39.044528 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:39.044090 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:18:39.044528 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:39.044168 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 06:18:39.092682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:39.092645 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 06:18:39.092960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:39.092930 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 06:18:40.044407 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:40.044371 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:18:40.044813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:40.044426 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 06:18:40.091821 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:40.091784 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 06:18:40.092125 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:40.092082 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 06:18:41.046316 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:41.044253 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 06:18:41.046316 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:41.044333 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 06:18:41.091771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:41.091736 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 06:18:41.092076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:41.092048 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 06:18:42.048922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:42.048893 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:42.049427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:42.049408 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:42.050054 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:42.050037 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:18:42.094508 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:42.094481 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:42.094865 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:42.094845 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:18:42.095461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:18:42.095441 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:19:03.702335 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:19:03.702274 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:19:08.244078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:08.244042 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk"] Oct 30 06:19:08.278977 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:08.278879 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6"] Oct 30 06:19:13.244382 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.244347 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" podUID="b0ea6618-0a2a-4142-99f2-83fdc378e095" containerName="payload-processing" containerID="cri-o://f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0" gracePeriod=25 Oct 30 06:19:13.279924 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.279864 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" podUID="3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0" containerName="payload-pre-processing" containerID="cri-o://ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510" gracePeriod=25 Oct 30 06:19:13.553085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.553057 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:19:13.585693 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.585671 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:19:13.603436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.603412 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ea6618-0a2a-4142-99f2-83fdc378e095-kube-api-access-47cds\" (UniqueName: \"kubernetes.io/projected/b0ea6618-0a2a-4142-99f2-83fdc378e095-kube-api-access-47cds\") pod \"b0ea6618-0a2a-4142-99f2-83fdc378e095\" (UID: \"b0ea6618-0a2a-4142-99f2-83fdc378e095\") " Oct 30 06:19:13.603577 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.603459 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ea6618-0a2a-4142-99f2-83fdc378e095-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ea6618-0a2a-4142-99f2-83fdc378e095-plugins-config-volume\") pod \"b0ea6618-0a2a-4142-99f2-83fdc378e095\" (UID: \"b0ea6618-0a2a-4142-99f2-83fdc378e095\") " Oct 30 06:19:13.603926 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.603901 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ea6618-0a2a-4142-99f2-83fdc378e095-plugins-config-volume" pod "b0ea6618-0a2a-4142-99f2-83fdc378e095" (UID: "b0ea6618-0a2a-4142-99f2-83fdc378e095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:13.604049 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.604024 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ea6618-0a2a-4142-99f2-83fdc378e095-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:19:13.605979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.605953 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ea6618-0a2a-4142-99f2-83fdc378e095-kube-api-access-47cds" pod "b0ea6618-0a2a-4142-99f2-83fdc378e095" (UID: "b0ea6618-0a2a-4142-99f2-83fdc378e095"). InnerVolumeSpecName "kube-api-access-47cds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:13.705351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.705289 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-plugins-config-volume\") pod \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\" (UID: \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\") " Oct 30 06:19:13.705559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.705437 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-kube-api-access-hn5rf\" (UniqueName: \"kubernetes.io/projected/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-kube-api-access-hn5rf\") pod \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\" (UID: \"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0\") " Oct 30 06:19:13.705720 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.705694 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-plugins-config-volume" pod "3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0" (UID: "3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:13.705855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.705839 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47cds\" (UniqueName: \"kubernetes.io/projected/b0ea6618-0a2a-4142-99f2-83fdc378e095-kube-api-access-47cds\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:19:13.705892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.705862 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:19:13.707635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.707606 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-kube-api-access-hn5rf" pod "3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0" (UID: "3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0"). InnerVolumeSpecName "kube-api-access-hn5rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:13.807670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.807591 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5rf\" (UniqueName: \"kubernetes.io/projected/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0-kube-api-access-hn5rf\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:19:13.936318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.936273 4431 generic.go:356] "Generic (PLEG): container finished" podID="b0ea6618-0a2a-4142-99f2-83fdc378e095" containerID="f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0" exitCode=0 Oct 30 06:19:13.936492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.936342 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" Oct 30 06:19:13.936492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.936342 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" event={"ID":"b0ea6618-0a2a-4142-99f2-83fdc378e095","Type":"ContainerDied","Data":"f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0"} Oct 30 06:19:13.936492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.936404 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk" event={"ID":"b0ea6618-0a2a-4142-99f2-83fdc378e095","Type":"ContainerDied","Data":"cd49740f6f3bdbdac81e496e6e5217a5528e075656980b0fc09ed0b5b2a22cd5"} Oct 30 06:19:13.936492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.936436 4431 scope.go:122] "RemoveContainer" containerID="f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0" Oct 30 06:19:13.938066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.938042 4431 generic.go:356] "Generic (PLEG): container finished" podID="3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0" containerID="ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510" exitCode=0 Oct 30 06:19:13.938195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.938088 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" Oct 30 06:19:13.938195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.938117 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" event={"ID":"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0","Type":"ContainerDied","Data":"ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510"} Oct 30 06:19:13.938195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.938157 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6" event={"ID":"3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0","Type":"ContainerDied","Data":"5b82eeb294e0657d7028aa2d3e5b9aab89100ea6af85e9a8439b9e8aeb6052c9"} Oct 30 06:19:13.947751 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.947732 4431 scope.go:122] "RemoveContainer" containerID="f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0" Oct 30 06:19:13.948031 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:19:13.948009 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0\": container with ID starting with f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0 not found: ID does not exist" containerID="f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0" Oct 30 06:19:13.948137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.948037 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0"} err="failed to get container status \"f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0\": rpc error: code = NotFound desc = could not find container \"f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0\": container with ID starting with f2a189a47891695762a0f36de713f4e7325d310c36db493d9c98a20562d5efc0 not found: ID does not exist" Oct 30 06:19:13.948137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.948056 4431 scope.go:122] "RemoveContainer" containerID="ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510" Oct 30 06:19:13.958282 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.958264 4431 scope.go:122] "RemoveContainer" containerID="ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510" Oct 30 06:19:13.958557 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:19:13.958540 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510\": container with ID starting with ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510 not found: ID does not exist" containerID="ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510" Oct 30 06:19:13.958629 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.958563 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510"} err="failed to get container status \"ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510\": rpc error: code = NotFound desc = could not find container \"ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510\": container with ID starting with ed25a99131862064340aade98369d71211d22f2152c7eb0fc18e537605c31510 not found: ID does not exist" Oct 30 06:19:13.988330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:13.988297 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6"] Oct 30 06:19:14.000257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:14.000239 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8272af-6c8fcd9cb7-9lnl6"] Oct 30 06:19:14.026847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:14.026571 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk"] Oct 30 06:19:14.029835 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:14.029814 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8272af-64c9997f4b-8kzwk"] Oct 30 06:19:15.338384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:15.338356 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0" path="/var/lib/kubelet/pods/3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0/volumes" Oct 30 06:19:15.338761 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:15.338733 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ea6618-0a2a-4142-99f2-83fdc378e095" path="/var/lib/kubelet/pods/b0ea6618-0a2a-4142-99f2-83fdc378e095/volumes" Oct 30 06:19:33.702578 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:19:33.702538 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:19:34.821733 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.821699 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p"] Oct 30 06:19:34.822266 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.822252 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0" containerName="payload-pre-processing" Oct 30 06:19:34.822326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.822267 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0" containerName="payload-pre-processing" Oct 30 06:19:34.822326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.822283 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b0ea6618-0a2a-4142-99f2-83fdc378e095" containerName="payload-processing" Oct 30 06:19:34.822326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.822288 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ea6618-0a2a-4142-99f2-83fdc378e095" containerName="payload-processing" Oct 30 06:19:34.822430 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.822393 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0171d1-c65f-4cef-a7a6-9d6e7b775bd0" containerName="payload-pre-processing" Oct 30 06:19:34.822430 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.822408 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ea6618-0a2a-4142-99f2-83fdc378e095" containerName="payload-processing" Oct 30 06:19:34.825783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.825764 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:34.845645 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.845623 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p"] Oct 30 06:19:34.853786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.853765 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f2cf8a\"" Oct 30 06:19:34.853944 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.853776 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f2cf8a-dockercfg-jqvjb\"" Oct 30 06:19:34.876563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.876534 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rdgs\" (UniqueName: \"kubernetes.io/projected/1af3a10e-4f37-4a89-8d49-d846fc00d964-kube-api-access-4rdgs\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p\" (UID: \"1af3a10e-4f37-4a89-8d49-d846fc00d964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:34.876838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.876817 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af3a10e-4f37-4a89-8d49-d846fc00d964-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p\" (UID: \"1af3a10e-4f37-4a89-8d49-d846fc00d964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:34.993323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.979106 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af3a10e-4f37-4a89-8d49-d846fc00d964-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p\" (UID: \"1af3a10e-4f37-4a89-8d49-d846fc00d964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:34.993323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.979217 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rdgs\" (UniqueName: \"kubernetes.io/projected/1af3a10e-4f37-4a89-8d49-d846fc00d964-kube-api-access-4rdgs\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p\" (UID: \"1af3a10e-4f37-4a89-8d49-d846fc00d964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:34.993323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:34.980532 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af3a10e-4f37-4a89-8d49-d846fc00d964-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p\" (UID: \"1af3a10e-4f37-4a89-8d49-d846fc00d964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:35.049718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:35.049687 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rdgs\" (UniqueName: \"kubernetes.io/projected/1af3a10e-4f37-4a89-8d49-d846fc00d964-kube-api-access-4rdgs\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p\" (UID: \"1af3a10e-4f37-4a89-8d49-d846fc00d964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:35.143704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:35.143611 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:35.463059 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:35.463032 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p"] Oct 30 06:19:35.464383 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:19:35.464359 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af3a10e_4f37_4a89_8d49_d846fc00d964.slice/crio-37e05fb9e36ede949fe33155b4b31fc5d47070418854de46e031430546d9745f WatchSource:0}: Error finding container 37e05fb9e36ede949fe33155b4b31fc5d47070418854de46e031430546d9745f: Status 404 returned error can't find the container with id 37e05fb9e36ede949fe33155b4b31fc5d47070418854de46e031430546d9745f Oct 30 06:19:36.085910 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:36.085876 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" event={"ID":"1af3a10e-4f37-4a89-8d49-d846fc00d964","Type":"ContainerStarted","Data":"621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6"} Oct 30 06:19:36.086295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:36.085928 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" event={"ID":"1af3a10e-4f37-4a89-8d49-d846fc00d964","Type":"ContainerStarted","Data":"37e05fb9e36ede949fe33155b4b31fc5d47070418854de46e031430546d9745f"} Oct 30 06:19:36.086295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:36.086001 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:36.128062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:36.128016 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" podStartSLOduration=2.128002241 podStartE2EDuration="2.128002241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:36.125114305 +0000 UTC m=+1584.931636054" watchObservedRunningTime="2026-10-30 06:19:36.128002241 +0000 UTC m=+1584.934523988" Oct 30 06:19:47.094356 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:47.094326 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:19:56.822454 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:56.822422 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr"] Oct 30 06:19:56.822952 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:56.822682 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" containerID="cri-o://e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33" gracePeriod=30 Oct 30 06:19:56.833473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:56.833447 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6"] Oct 30 06:19:56.833651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:56.833631 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" containerID="cri-o://9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a" gracePeriod=30 Oct 30 06:19:56.842255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:56.842234 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq"] Oct 30 06:19:56.842451 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:56.842431 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" containerID="cri-o://be581c896c31a190e0a1ab0a51358d8467803595cb079316ac05b62cdfe59927" gracePeriod=30 Oct 30 06:19:56.908602 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:56.908470 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:56.908892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:56.908865 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:57.051937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:57.051892 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:57.052125 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:57.051963 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:57.097601 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:57.097530 4431 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:57.097601 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:19:57.097590 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:20:01.919667 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:20:01.919634 4431 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd750fc01_847b_4584_8ec9_b039222088ee.slice/crio-be581c896c31a190e0a1ab0a51358d8467803595cb079316ac05b62cdfe59927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd750fc01_847b_4584_8ec9_b039222088ee.slice/crio-conmon-be581c896c31a190e0a1ab0a51358d8467803595cb079316ac05b62cdfe59927.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:20:01.920046 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:20:01.919634 4431 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd750fc01_847b_4584_8ec9_b039222088ee.slice/crio-be581c896c31a190e0a1ab0a51358d8467803595cb079316ac05b62cdfe59927.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd750fc01_847b_4584_8ec9_b039222088ee.slice/crio-conmon-be581c896c31a190e0a1ab0a51358d8467803595cb079316ac05b62cdfe59927.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:20:02.204018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.203680 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:20:02.224637 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.223695 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:20:02.279632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.279586 4431 generic.go:356] "Generic (PLEG): container finished" podID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerID="9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a" exitCode=0 Oct 30 06:20:02.279904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.279876 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" event={"ID":"6964007e-22ee-4328-b525-7fc9cbb2ebea","Type":"ContainerDied","Data":"9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a"} Oct 30 06:20:02.280024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.280004 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" event={"ID":"6964007e-22ee-4328-b525-7fc9cbb2ebea","Type":"ContainerDied","Data":"ec89e0d33090ad5a6f443365cfb13093b1cca138d22eafc215014ce283b58f7d"} Oct 30 06:20:02.280091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.280059 4431 scope.go:122] "RemoveContainer" containerID="9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a" Oct 30 06:20:02.280337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.280285 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6" Oct 30 06:20:02.284207 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.284171 4431 generic.go:356] "Generic (PLEG): container finished" podID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerID="e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33" exitCode=0 Oct 30 06:20:02.284339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.284273 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" event={"ID":"e8117705-4f64-42fe-93e5-1425a3e1087f","Type":"ContainerDied","Data":"e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33"} Oct 30 06:20:02.284408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.284353 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" event={"ID":"e8117705-4f64-42fe-93e5-1425a3e1087f","Type":"ContainerDied","Data":"476dd96596b942a04c6f13f69335f0bcdcf3e0bc4d4af8ad0386e84d024d3f3c"} Oct 30 06:20:02.284408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.284373 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr" Oct 30 06:20:02.286710 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.286681 4431 generic.go:356] "Generic (PLEG): container finished" podID="d750fc01-847b-4584-8ec9-b039222088ee" containerID="be581c896c31a190e0a1ab0a51358d8467803595cb079316ac05b62cdfe59927" exitCode=0 Oct 30 06:20:02.286846 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.286728 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" event={"ID":"d750fc01-847b-4584-8ec9-b039222088ee","Type":"ContainerDied","Data":"be581c896c31a190e0a1ab0a51358d8467803595cb079316ac05b62cdfe59927"} Oct 30 06:20:02.288344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.288325 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:20:02.296082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.296066 4431 scope.go:122] "RemoveContainer" containerID="9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a" Oct 30 06:20:02.296374 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:20:02.296351 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a\": container with ID starting with 9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a not found: ID does not exist" containerID="9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a" Oct 30 06:20:02.296455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.296379 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a"} err="failed to get container status \"9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a\": rpc error: code = NotFound desc = could not find container \"9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a\": container with ID starting with 9fe28e2d573b37be740f0f791864df16b071692ec6d70ff65974b6bc2cedb82a not found: ID does not exist" Oct 30 06:20:02.296455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.296394 4431 scope.go:122] "RemoveContainer" containerID="e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33" Oct 30 06:20:02.305718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.305691 4431 scope.go:122] "RemoveContainer" containerID="e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33" Oct 30 06:20:02.305974 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:20:02.305951 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33\": container with ID starting with e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33 not found: ID does not exist" containerID="e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33" Oct 30 06:20:02.306057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.305983 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33"} err="failed to get container status \"e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33\": rpc error: code = NotFound desc = could not find container \"e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33\": container with ID starting with e7a37f159a5c890d2b8f19ee3df807db6c36dbaf14d14b27a7bc393270d5cb33 not found: ID does not exist" Oct 30 06:20:02.333778 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.333749 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-data\") pod \"e8117705-4f64-42fe-93e5-1425a3e1087f\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " Oct 30 06:20:02.333932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.333804 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-socket\") pod \"6964007e-22ee-4328-b525-7fc9cbb2ebea\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " Oct 30 06:20:02.333932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.333824 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-token\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-token\") pod \"e8117705-4f64-42fe-93e5-1425a3e1087f\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " Oct 30 06:20:02.333932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.333843 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8117705-4f64-42fe-93e5-1425a3e1087f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8117705-4f64-42fe-93e5-1425a3e1087f-istiod-ca-cert\") pod \"e8117705-4f64-42fe-93e5-1425a3e1087f\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " Oct 30 06:20:02.333932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.333865 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-data\") pod \"6964007e-22ee-4328-b525-7fc9cbb2ebea\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " Oct 30 06:20:02.333932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.333903 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-certs\") pod \"6964007e-22ee-4328-b525-7fc9cbb2ebea\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " Oct 30 06:20:02.333932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.333921 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-socket\") pod \"e8117705-4f64-42fe-93e5-1425a3e1087f\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " Oct 30 06:20:02.334269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.333939 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-credential-socket\") pod \"e8117705-4f64-42fe-93e5-1425a3e1087f\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " Oct 30 06:20:02.334269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.333984 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-kube-api-access-jvqjq\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-kube-api-access-jvqjq\") pod \"e8117705-4f64-42fe-93e5-1425a3e1087f\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " Oct 30 06:20:02.334269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334001 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-token\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-token\") pod \"6964007e-22ee-4328-b525-7fc9cbb2ebea\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " Oct 30 06:20:02.334269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334026 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-podinfo\") pod \"6964007e-22ee-4328-b525-7fc9cbb2ebea\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " Oct 30 06:20:02.334269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334049 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-certs\") pod \"e8117705-4f64-42fe-93e5-1425a3e1087f\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " Oct 30 06:20:02.334269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334138 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-credential-socket\") pod \"6964007e-22ee-4328-b525-7fc9cbb2ebea\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " Oct 30 06:20:02.334269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334187 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6964007e-22ee-4328-b525-7fc9cbb2ebea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6964007e-22ee-4328-b525-7fc9cbb2ebea-istiod-ca-cert\") pod \"6964007e-22ee-4328-b525-7fc9cbb2ebea\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " Oct 30 06:20:02.334269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334207 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-envoy\") pod \"6964007e-22ee-4328-b525-7fc9cbb2ebea\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " Oct 30 06:20:02.334269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334266 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-envoy\") pod \"e8117705-4f64-42fe-93e5-1425a3e1087f\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " Oct 30 06:20:02.334863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334286 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-podinfo\") pod \"e8117705-4f64-42fe-93e5-1425a3e1087f\" (UID: \"e8117705-4f64-42fe-93e5-1425a3e1087f\") " Oct 30 06:20:02.334863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334396 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-kube-api-access-hzt6n\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-kube-api-access-hzt6n\") pod \"6964007e-22ee-4328-b525-7fc9cbb2ebea\" (UID: \"6964007e-22ee-4328-b525-7fc9cbb2ebea\") " Oct 30 06:20:02.334863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334606 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-credential-socket" pod "e8117705-4f64-42fe-93e5-1425a3e1087f" (UID: "e8117705-4f64-42fe-93e5-1425a3e1087f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.334863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334755 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-socket" pod "6964007e-22ee-4328-b525-7fc9cbb2ebea" (UID: "6964007e-22ee-4328-b525-7fc9cbb2ebea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.334863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.334846 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-data" pod "6964007e-22ee-4328-b525-7fc9cbb2ebea" (UID: "6964007e-22ee-4328-b525-7fc9cbb2ebea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.335176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.335079 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.335176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.335102 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.335176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.335103 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-data" pod "e8117705-4f64-42fe-93e5-1425a3e1087f" (UID: "e8117705-4f64-42fe-93e5-1425a3e1087f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.335176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.335116 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.335434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.335358 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-certs" pod "6964007e-22ee-4328-b525-7fc9cbb2ebea" (UID: "6964007e-22ee-4328-b525-7fc9cbb2ebea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.335567 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.335543 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-socket" pod "e8117705-4f64-42fe-93e5-1425a3e1087f" (UID: "e8117705-4f64-42fe-93e5-1425a3e1087f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.336168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.336140 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6964007e-22ee-4328-b525-7fc9cbb2ebea-istiod-ca-cert" pod "6964007e-22ee-4328-b525-7fc9cbb2ebea" (UID: "6964007e-22ee-4328-b525-7fc9cbb2ebea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.336712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.336685 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8117705-4f64-42fe-93e5-1425a3e1087f-istiod-ca-cert" pod "e8117705-4f64-42fe-93e5-1425a3e1087f" (UID: "e8117705-4f64-42fe-93e5-1425a3e1087f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.337926 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.337898 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-token" pod "e8117705-4f64-42fe-93e5-1425a3e1087f" (UID: "e8117705-4f64-42fe-93e5-1425a3e1087f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.338151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.338132 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-kube-api-access-jvqjq" pod "e8117705-4f64-42fe-93e5-1425a3e1087f" (UID: "e8117705-4f64-42fe-93e5-1425a3e1087f"). InnerVolumeSpecName "kube-api-access-jvqjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.338281 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.338266 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-credential-socket" pod "6964007e-22ee-4328-b525-7fc9cbb2ebea" (UID: "6964007e-22ee-4328-b525-7fc9cbb2ebea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.338537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.338513 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-certs" pod "e8117705-4f64-42fe-93e5-1425a3e1087f" (UID: "e8117705-4f64-42fe-93e5-1425a3e1087f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.343669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.343612 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-envoy" pod "e8117705-4f64-42fe-93e5-1425a3e1087f" (UID: "e8117705-4f64-42fe-93e5-1425a3e1087f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.343669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.343615 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-envoy" pod "6964007e-22ee-4328-b525-7fc9cbb2ebea" (UID: "6964007e-22ee-4328-b525-7fc9cbb2ebea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.343669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.343635 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-podinfo" pod "e8117705-4f64-42fe-93e5-1425a3e1087f" (UID: "e8117705-4f64-42fe-93e5-1425a3e1087f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.343873 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.343664 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-token" pod "6964007e-22ee-4328-b525-7fc9cbb2ebea" (UID: "6964007e-22ee-4328-b525-7fc9cbb2ebea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.343873 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.343681 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-kube-api-access-hzt6n" pod "6964007e-22ee-4328-b525-7fc9cbb2ebea" (UID: "6964007e-22ee-4328-b525-7fc9cbb2ebea"). InnerVolumeSpecName "kube-api-access-hzt6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.345149 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.345126 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-podinfo" pod "6964007e-22ee-4328-b525-7fc9cbb2ebea" (UID: "6964007e-22ee-4328-b525-7fc9cbb2ebea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.435583 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.435492 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-certs\") pod \"d750fc01-847b-4584-8ec9-b039222088ee\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " Oct 30 06:20:02.435752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.435588 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-data\") pod \"d750fc01-847b-4584-8ec9-b039222088ee\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " Oct 30 06:20:02.435752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.435744 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d750fc01-847b-4584-8ec9-b039222088ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750fc01-847b-4584-8ec9-b039222088ee-istiod-ca-cert\") pod \"d750fc01-847b-4584-8ec9-b039222088ee\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " Oct 30 06:20:02.435867 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.435809 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-istio-token\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-istio-token\") pod \"d750fc01-847b-4584-8ec9-b039222088ee\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " Oct 30 06:20:02.435867 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.435841 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-certs" pod "d750fc01-847b-4584-8ec9-b039222088ee" (UID: "d750fc01-847b-4584-8ec9-b039222088ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.435867 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.435859 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-credential-socket\") pod \"d750fc01-847b-4584-8ec9-b039222088ee\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " Oct 30 06:20:02.436024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.435875 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-data" pod "d750fc01-847b-4584-8ec9-b039222088ee" (UID: "d750fc01-847b-4584-8ec9-b039222088ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.436024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.435913 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-envoy\") pod \"d750fc01-847b-4584-8ec9-b039222088ee\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " Oct 30 06:20:02.436024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.435942 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d750fc01-847b-4584-8ec9-b039222088ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750fc01-847b-4584-8ec9-b039222088ee-istio-podinfo\") pod \"d750fc01-847b-4584-8ec9-b039222088ee\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " Oct 30 06:20:02.436172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436023 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-socket\") pod \"d750fc01-847b-4584-8ec9-b039222088ee\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " Oct 30 06:20:02.436172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436071 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-kube-api-access-vmqvs\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-kube-api-access-vmqvs\") pod \"d750fc01-847b-4584-8ec9-b039222088ee\" (UID: \"d750fc01-847b-4584-8ec9-b039222088ee\") " Oct 30 06:20:02.436172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436101 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-credential-socket" pod "d750fc01-847b-4584-8ec9-b039222088ee" (UID: "d750fc01-847b-4584-8ec9-b039222088ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.436172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436134 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d750fc01-847b-4584-8ec9-b039222088ee-istiod-ca-cert" pod "d750fc01-847b-4584-8ec9-b039222088ee" (UID: "d750fc01-847b-4584-8ec9-b039222088ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436705 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvqjq\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-kube-api-access-jvqjq\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436731 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436746 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436761 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436819 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436834 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436846 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436859 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6964007e-22ee-4328-b525-7fc9cbb2ebea-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436874 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436888 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436890 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-socket" pod "d750fc01-847b-4584-8ec9-b039222088ee" (UID: "d750fc01-847b-4584-8ec9-b039222088ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436908 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436924 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzt6n\" (UniqueName: \"kubernetes.io/projected/6964007e-22ee-4328-b525-7fc9cbb2ebea-kube-api-access-hzt6n\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436937 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436950 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d750fc01-847b-4584-8ec9-b039222088ee-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436962 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8117705-4f64-42fe-93e5-1425a3e1087f-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436973 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8117705-4f64-42fe-93e5-1425a3e1087f-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436987 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.436999 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6964007e-22ee-4328-b525-7fc9cbb2ebea-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.437249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.437013 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8117705-4f64-42fe-93e5-1425a3e1087f-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.439271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.439233 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-istio-token" pod "d750fc01-847b-4584-8ec9-b039222088ee" (UID: "d750fc01-847b-4584-8ec9-b039222088ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.439599 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.439567 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-envoy" pod "d750fc01-847b-4584-8ec9-b039222088ee" (UID: "d750fc01-847b-4584-8ec9-b039222088ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.439948 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.439929 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d750fc01-847b-4584-8ec9-b039222088ee-istio-podinfo" pod "d750fc01-847b-4584-8ec9-b039222088ee" (UID: "d750fc01-847b-4584-8ec9-b039222088ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.440026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.439984 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-kube-api-access-vmqvs" pod "d750fc01-847b-4584-8ec9-b039222088ee" (UID: "d750fc01-847b-4584-8ec9-b039222088ee"). InnerVolumeSpecName "kube-api-access-vmqvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.538233 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.538196 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.538233 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.538229 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.538438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.538243 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d750fc01-847b-4584-8ec9-b039222088ee-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.538438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.538255 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d750fc01-847b-4584-8ec9-b039222088ee-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.538438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.538268 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmqvs\" (UniqueName: \"kubernetes.io/projected/d750fc01-847b-4584-8ec9-b039222088ee-kube-api-access-vmqvs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.630341 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.630315 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6"] Oct 30 06:20:02.636667 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.636636 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-2c5n6"] Oct 30 06:20:02.669026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.669001 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr"] Oct 30 06:20:02.675511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.675484 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8kwrr"] Oct 30 06:20:02.695945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.695910 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f"] Oct 30 06:20:02.696533 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.696519 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" Oct 30 06:20:02.696574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.696534 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" Oct 30 06:20:02.696574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.696550 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" Oct 30 06:20:02.696574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.696555 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" Oct 30 06:20:02.696671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.696579 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" Oct 30 06:20:02.696671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.696586 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" Oct 30 06:20:02.696731 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.696677 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d750fc01-847b-4584-8ec9-b039222088ee" containerName="istio-proxy" Oct 30 06:20:02.696731 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.696689 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" containerName="istio-proxy" Oct 30 06:20:02.696731 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.696696 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" containerName="istio-proxy" Oct 30 06:20:02.701505 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.701488 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.718924 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.718904 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f2cf8a-serving-cert\"" Oct 30 06:20:02.721163 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.721141 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f"] Oct 30 06:20:02.842044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.842007 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrg4x\" (UniqueName: \"kubernetes.io/projected/0af70ae2-6fdb-455d-89f8-024f6003f52e-kube-api-access-zrg4x\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.842234 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.842213 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.842383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.842366 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-metrics-certs\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.842434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.842404 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af70ae2-6fdb-455d-89f8-024f6003f52e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.946499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.944479 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.946499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.944616 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-metrics-certs\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.946499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.944661 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af70ae2-6fdb-455d-89f8-024f6003f52e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.946499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.944734 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrg4x\" (UniqueName: \"kubernetes.io/projected/0af70ae2-6fdb-455d-89f8-024f6003f52e-kube-api-access-zrg4x\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.947611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.946508 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af70ae2-6fdb-455d-89f8-024f6003f52e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.948334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.948282 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.964325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.949247 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-metrics-certs\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:02.972063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:02.972036 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrg4x\" (UniqueName: \"kubernetes.io/projected/0af70ae2-6fdb-455d-89f8-024f6003f52e-kube-api-access-zrg4x\") pod \"maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:03.015083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.015055 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:03.296099 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.296024 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" Oct 30 06:20:03.296099 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.296052 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq" event={"ID":"d750fc01-847b-4584-8ec9-b039222088ee","Type":"ContainerDied","Data":"4014303935cd389bdcbc075ce2ba06b3f7e7f23cc75e2529cf80ef79cb85f8e0"} Oct 30 06:20:03.296370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.296111 4431 scope.go:122] "RemoveContainer" containerID="be581c896c31a190e0a1ab0a51358d8467803595cb079316ac05b62cdfe59927" Oct 30 06:20:03.321436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.321414 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f"] Oct 30 06:20:03.321672 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:20:03.321653 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af70ae2_6fdb_455d_89f8_024f6003f52e.slice/crio-60a3fcf34593d720eb0be6d947288e6052ae70b4dfd1dc7059a4dd793602fd1c WatchSource:0}: Error finding container 60a3fcf34593d720eb0be6d947288e6052ae70b4dfd1dc7059a4dd793602fd1c: Status 404 returned error can't find the container with id 60a3fcf34593d720eb0be6d947288e6052ae70b4dfd1dc7059a4dd793602fd1c Oct 30 06:20:03.322817 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.322801 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:20:03.338752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.338726 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6964007e-22ee-4328-b525-7fc9cbb2ebea" path="/var/lib/kubelet/pods/6964007e-22ee-4328-b525-7fc9cbb2ebea/volumes" Oct 30 06:20:03.339290 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.339270 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8117705-4f64-42fe-93e5-1425a3e1087f" path="/var/lib/kubelet/pods/e8117705-4f64-42fe-93e5-1425a3e1087f/volumes" Oct 30 06:20:03.348823 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.348802 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq"] Oct 30 06:20:03.354866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:03.354848 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-gwqtq"] Oct 30 06:20:03.701930 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:20:03.701899 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:20:04.306124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:04.306085 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" event={"ID":"0af70ae2-6fdb-455d-89f8-024f6003f52e","Type":"ContainerStarted","Data":"c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9"} Oct 30 06:20:04.306564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:04.306148 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" event={"ID":"0af70ae2-6fdb-455d-89f8-024f6003f52e","Type":"ContainerStarted","Data":"60a3fcf34593d720eb0be6d947288e6052ae70b4dfd1dc7059a4dd793602fd1c"} Oct 30 06:20:04.306564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:04.306292 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:04.347993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:04.347943 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" podStartSLOduration=1.9474605440000001 podStartE2EDuration="2.347929894s" totalImagesPullingTime="400.46935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:04.347633558 +0000 UTC m=+1613.154155306" watchObservedRunningTime="2026-10-30 06:20:04.347929894 +0000 UTC m=+1613.154451696" Oct 30 06:20:05.337878 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:05.337838 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d750fc01-847b-4584-8ec9-b039222088ee" path="/var/lib/kubelet/pods/d750fc01-847b-4584-8ec9-b039222088ee/volumes" Oct 30 06:20:10.318431 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:10.318399 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:20:33.702265 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:20:33.702223 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:20:34.171658 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:34.171549 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p"] Oct 30 06:20:39.172769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.172716 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" podUID="1af3a10e-4f37-4a89-8d49-d846fc00d964" containerName="payload-pre-processing" containerID="cri-o://621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6" gracePeriod=25 Oct 30 06:20:39.444102 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.444067 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:20:39.583867 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.583827 4431 generic.go:356] "Generic (PLEG): container finished" podID="1af3a10e-4f37-4a89-8d49-d846fc00d964" containerID="621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6" exitCode=0 Oct 30 06:20:39.584057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.583885 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" Oct 30 06:20:39.584057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.583916 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" event={"ID":"1af3a10e-4f37-4a89-8d49-d846fc00d964","Type":"ContainerDied","Data":"621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6"} Oct 30 06:20:39.584057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.583974 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p" event={"ID":"1af3a10e-4f37-4a89-8d49-d846fc00d964","Type":"ContainerDied","Data":"37e05fb9e36ede949fe33155b4b31fc5d47070418854de46e031430546d9745f"} Oct 30 06:20:39.584057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.583997 4431 scope.go:122] "RemoveContainer" containerID="621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6" Oct 30 06:20:39.594801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.594776 4431 scope.go:122] "RemoveContainer" containerID="621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6" Oct 30 06:20:39.595113 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:20:39.595086 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6\": container with ID starting with 621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6 not found: ID does not exist" containerID="621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6" Oct 30 06:20:39.595193 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.595122 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6"} err="failed to get container status \"621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6\": rpc error: code = NotFound desc = could not find container \"621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6\": container with ID starting with 621b1a411c4a493272e79b4fb3fc3dd1eca596bc9b8655c699fdd451f3d624d6 not found: ID does not exist" Oct 30 06:20:39.613982 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.613960 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af3a10e-4f37-4a89-8d49-d846fc00d964-kube-api-access-4rdgs\" (UniqueName: \"kubernetes.io/projected/1af3a10e-4f37-4a89-8d49-d846fc00d964-kube-api-access-4rdgs\") pod \"1af3a10e-4f37-4a89-8d49-d846fc00d964\" (UID: \"1af3a10e-4f37-4a89-8d49-d846fc00d964\") " Oct 30 06:20:39.614165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.614151 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af3a10e-4f37-4a89-8d49-d846fc00d964-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af3a10e-4f37-4a89-8d49-d846fc00d964-plugins-config-volume\") pod \"1af3a10e-4f37-4a89-8d49-d846fc00d964\" (UID: \"1af3a10e-4f37-4a89-8d49-d846fc00d964\") " Oct 30 06:20:39.614550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.614526 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af3a10e-4f37-4a89-8d49-d846fc00d964-plugins-config-volume" pod "1af3a10e-4f37-4a89-8d49-d846fc00d964" (UID: "1af3a10e-4f37-4a89-8d49-d846fc00d964"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:39.616175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.616153 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af3a10e-4f37-4a89-8d49-d846fc00d964-kube-api-access-4rdgs" pod "1af3a10e-4f37-4a89-8d49-d846fc00d964" (UID: "1af3a10e-4f37-4a89-8d49-d846fc00d964"). InnerVolumeSpecName "kube-api-access-4rdgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:39.716195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.716159 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rdgs\" (UniqueName: \"kubernetes.io/projected/1af3a10e-4f37-4a89-8d49-d846fc00d964-kube-api-access-4rdgs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:39.716195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.716189 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af3a10e-4f37-4a89-8d49-d846fc00d964-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:20:39.929421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.929379 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p"] Oct 30 06:20:39.942869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:39.942832 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-7f489f7795-9lp9p"] Oct 30 06:20:41.342427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:20:41.342394 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af3a10e-4f37-4a89-8d49-d846fc00d964" path="/var/lib/kubelet/pods/1af3a10e-4f37-4a89-8d49-d846fc00d964/volumes" Oct 30 06:21:03.702840 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:21:03.702742 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:21:08.958975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:08.958946 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hpphs_9b2cce36-89ca-4ea5-a488-be5af53dbf14/payload-processing/0.log" Oct 30 06:21:09.204019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.203992 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hpphs_9b2cce36-89ca-4ea5-a488-be5af53dbf14/payload-processing/0.log" Oct 30 06:21:09.347339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.347231 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8"] Oct 30 06:21:09.347555 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.347511 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" podUID="fb45dcde-f629-4717-ad1c-66ad3bb86ac3" containerName="maas-api" containerID="cri-o://2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33" gracePeriod=30 Oct 30 06:21:09.629981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.629954 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:21:09.760758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.760719 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-openshift-service-ca\") pod \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " Oct 30 06:21:09.760965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.760890 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-maas-api-tls\") pod \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " Oct 30 06:21:09.760965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.760931 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-metrics-certs\") pod \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " Oct 30 06:21:09.761088 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.760989 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-kube-api-access-zbgql\" (UniqueName: \"kubernetes.io/projected/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-kube-api-access-zbgql\") pod \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\" (UID: \"fb45dcde-f629-4717-ad1c-66ad3bb86ac3\") " Oct 30 06:21:09.761138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.761078 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-openshift-service-ca" pod "fb45dcde-f629-4717-ad1c-66ad3bb86ac3" (UID: "fb45dcde-f629-4717-ad1c-66ad3bb86ac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:09.761471 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.761453 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.763172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.763140 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-metrics-certs" pod "fb45dcde-f629-4717-ad1c-66ad3bb86ac3" (UID: "fb45dcde-f629-4717-ad1c-66ad3bb86ac3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:09.763172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.763158 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-maas-api-tls" pod "fb45dcde-f629-4717-ad1c-66ad3bb86ac3" (UID: "fb45dcde-f629-4717-ad1c-66ad3bb86ac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:09.763338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.763204 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-kube-api-access-zbgql" pod "fb45dcde-f629-4717-ad1c-66ad3bb86ac3" (UID: "fb45dcde-f629-4717-ad1c-66ad3bb86ac3"). InnerVolumeSpecName "kube-api-access-zbgql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:09.775647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.775624 4431 generic.go:356] "Generic (PLEG): container finished" podID="fb45dcde-f629-4717-ad1c-66ad3bb86ac3" containerID="2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33" exitCode=0 Oct 30 06:21:09.775765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.775684 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" Oct 30 06:21:09.775765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.775713 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" event={"ID":"fb45dcde-f629-4717-ad1c-66ad3bb86ac3","Type":"ContainerDied","Data":"2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33"} Oct 30 06:21:09.775889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.775782 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8" event={"ID":"fb45dcde-f629-4717-ad1c-66ad3bb86ac3","Type":"ContainerDied","Data":"cd0d8c951c699309162311456f8d914d2eef6167c13f8e8eca0e10d2d1b33b7c"} Oct 30 06:21:09.775889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.775818 4431 scope.go:122] "RemoveContainer" containerID="2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33" Oct 30 06:21:09.796817 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.796799 4431 scope.go:122] "RemoveContainer" containerID="2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33" Oct 30 06:21:09.797152 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:21:09.797133 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33\": container with ID starting with 2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33 not found: ID does not exist" containerID="2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33" Oct 30 06:21:09.797204 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.797160 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33"} err="failed to get container status \"2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33\": rpc error: code = NotFound desc = could not find container \"2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33\": container with ID starting with 2da5b2a2a1e28f6185b5d42625c99743612f06bc39414c35eab5f3d02a562c33 not found: ID does not exist" Oct 30 06:21:09.844325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.836234 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8"] Oct 30 06:21:09.865383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.862899 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.865383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.862932 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.865383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.862949 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgql\" (UniqueName: \"kubernetes.io/projected/fb45dcde-f629-4717-ad1c-66ad3bb86ac3-kube-api-access-zbgql\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.883333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:09.882783 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a55a4b-5fb9d45dfb-64cn8"] Oct 30 06:21:11.339221 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:21:11.339188 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb45dcde-f629-4717-ad1c-66ad3bb86ac3" path="/var/lib/kubelet/pods/fb45dcde-f629-4717-ad1c-66ad3bb86ac3/volumes" Oct 30 06:21:33.703046 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:21:33.703004 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:22:03.702064 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:22:03.702028 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:22:21.514178 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:21.514138 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667"] Oct 30 06:22:21.514672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:21.514375 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" containerID="cri-o://0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804" gracePeriod=30 Oct 30 06:22:26.801006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.800973 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:22:26.951129 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951093 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istiod-ca-cert\") pod \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " Oct 30 06:22:26.951372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951156 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-credential-socket\") pod \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " Oct 30 06:22:26.951372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951201 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-kube-api-access-r5spr\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-kube-api-access-r5spr\") pod \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " Oct 30 06:22:26.951372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951255 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-certs\") pod \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " Oct 30 06:22:26.951372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951287 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-data\") pod \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " Oct 30 06:22:26.951372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951362 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-envoy\") pod \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " Oct 30 06:22:26.951651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951411 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-socket\") pod \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " Oct 30 06:22:26.951651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951460 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-token\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-token\") pod \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " Oct 30 06:22:26.951651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951599 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istiod-ca-cert" pod "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" (UID: "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:26.951651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951612 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-podinfo\") pod \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\" (UID: \"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4\") " Oct 30 06:22:26.951897 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951819 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-data" pod "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" (UID: "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.951897 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951882 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-certs" pod "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" (UID: "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.952025 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.951868 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-credential-socket" pod "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" (UID: "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.952086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.952070 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-socket" pod "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" (UID: "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.952480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.952419 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.952480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.952440 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.952480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.952455 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.952480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.952469 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.952480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.952483 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.954267 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.954234 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-podinfo" pod "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" (UID: "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:26.954511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.954486 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-kube-api-access-r5spr" pod "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" (UID: "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4"). InnerVolumeSpecName "kube-api-access-r5spr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:26.954999 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.954974 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-envoy" pod "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" (UID: "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.955251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:26.955231 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-token" pod "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" (UID: "4ffcbc2a-ddbb-40a1-b866-330d0a6234d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:27.053706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.053556 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5spr\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-kube-api-access-r5spr\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:27.053706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.053587 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:27.053706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.053596 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:27.053706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.053605 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:27.305160 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.305064 4431 generic.go:356] "Generic (PLEG): container finished" podID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerID="0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804" exitCode=0 Oct 30 06:22:27.305160 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.305122 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" Oct 30 06:22:27.305160 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.305145 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" event={"ID":"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4","Type":"ContainerDied","Data":"0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804"} Oct 30 06:22:27.305455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.305200 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667" event={"ID":"4ffcbc2a-ddbb-40a1-b866-330d0a6234d4","Type":"ContainerDied","Data":"1a89da679f178be84e6ccef023acc2012568a3ce9693fd975a992ad218b99595"} Oct 30 06:22:27.305455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.305223 4431 scope.go:122] "RemoveContainer" containerID="0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804" Oct 30 06:22:27.316368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.316352 4431 scope.go:122] "RemoveContainer" containerID="0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804" Oct 30 06:22:27.316642 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:22:27.316623 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804\": container with ID starting with 0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804 not found: ID does not exist" containerID="0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804" Oct 30 06:22:27.316701 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.316650 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804"} err="failed to get container status \"0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804\": rpc error: code = NotFound desc = could not find container \"0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804\": container with ID starting with 0d480263de9bb99e8857689c5a3a4fa054f13f36bb1f5adaa1db0f2d6b689804 not found: ID does not exist" Oct 30 06:22:27.361966 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.361929 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667"] Oct 30 06:22:27.374090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:27.374055 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-6s667"] Oct 30 06:22:29.338577 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:29.338540 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" path="/var/lib/kubelet/pods/4ffcbc2a-ddbb-40a1-b866-330d0a6234d4/volumes" Oct 30 06:22:33.700675 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:22:33.700644 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:22:34.320488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.320450 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq"] Oct 30 06:22:34.321234 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.321214 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fb45dcde-f629-4717-ad1c-66ad3bb86ac3" containerName="maas-api" Oct 30 06:22:34.321234 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.321232 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb45dcde-f629-4717-ad1c-66ad3bb86ac3" containerName="maas-api" Oct 30 06:22:34.321437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.321246 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1af3a10e-4f37-4a89-8d49-d846fc00d964" containerName="payload-pre-processing" Oct 30 06:22:34.321437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.321254 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af3a10e-4f37-4a89-8d49-d846fc00d964" containerName="payload-pre-processing" Oct 30 06:22:34.321437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.321270 4431 cpu_manager.go:414] "containerMap: removing container" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" Oct 30 06:22:34.321437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.321278 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" Oct 30 06:22:34.321437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.321422 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb45dcde-f629-4717-ad1c-66ad3bb86ac3" containerName="maas-api" Oct 30 06:22:34.321437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.321435 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af3a10e-4f37-4a89-8d49-d846fc00d964" containerName="payload-pre-processing" Oct 30 06:22:34.321750 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.321447 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ffcbc2a-ddbb-40a1-b866-330d0a6234d4" containerName="istio-proxy" Oct 30 06:22:34.325144 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.325124 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.339974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.339950 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-certs\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.340094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.339981 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghf9\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-kube-api-access-dghf9\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.340094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.340021 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-data\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.340196 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.340144 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-token\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.340196 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.340178 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-socket\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.340278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.340214 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-credential-socket\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.340338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.340281 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab613a-ebe8-40c9-88f6-559d74969539-istiod-ca-cert\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.340452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.340430 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-envoy\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.340578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.340560 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-podinfo\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.345161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.345125 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a035a3-openshift-default-dockercfg-vf5z4\"" Oct 30 06:22:34.359651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.359619 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq"] Oct 30 06:22:34.441511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.441475 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-envoy\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.441702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.441564 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-podinfo\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.441702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.441615 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-certs\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.441702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.441637 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghf9\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-kube-api-access-dghf9\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.441702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.441689 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-data\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.441941 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.441792 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-token\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.441941 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.441825 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-socket\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.441941 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.441866 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-credential-socket\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.441941 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.441927 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab613a-ebe8-40c9-88f6-559d74969539-istiod-ca-cert\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.442153 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.442012 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-certs\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.442386 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.442364 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-data\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.442488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.442446 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-socket\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.442574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.442551 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-credential-socket\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.442702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.442684 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab613a-ebe8-40c9-88f6-559d74969539-istiod-ca-cert\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.444434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.444414 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-envoy\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.444625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.444606 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-podinfo\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.493454 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.493415 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghf9\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-kube-api-access-dghf9\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.519673 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.519634 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-token\") pod \"e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.638406 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.638291 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:34.984440 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.984404 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq"] Oct 30 06:22:34.990896 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:22:34.990863 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ab613a_ebe8_40c9_88f6_559d74969539.slice/crio-f4c5b7114c1f8c87b15f57ad3308f2a43f0134e4158b9e3e6c6cdf1e8e4312ad WatchSource:0}: Error finding container f4c5b7114c1f8c87b15f57ad3308f2a43f0134e4158b9e3e6c6cdf1e8e4312ad: Status 404 returned error can't find the container with id f4c5b7114c1f8c87b15f57ad3308f2a43f0134e4158b9e3e6c6cdf1e8e4312ad Oct 30 06:22:34.993776 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.993736 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:22:34.993875 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.993833 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:22:34.993916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:34.993880 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:22:35.355918 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:35.355815 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" event={"ID":"a7ab613a-ebe8-40c9-88f6-559d74969539","Type":"ContainerStarted","Data":"2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17"} Oct 30 06:22:35.355918 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:35.355870 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" event={"ID":"a7ab613a-ebe8-40c9-88f6-559d74969539","Type":"ContainerStarted","Data":"f4c5b7114c1f8c87b15f57ad3308f2a43f0134e4158b9e3e6c6cdf1e8e4312ad"} Oct 30 06:22:35.408931 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:35.408860 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" podStartSLOduration=1.408837581 podStartE2EDuration="1.408837581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:35.405168651 +0000 UTC m=+1764.211690485" watchObservedRunningTime="2026-10-30 06:22:35.408837581 +0000 UTC m=+1764.215359330" Oct 30 06:22:35.639415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:35.639335 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:36.566047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:36.565989 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh"] Oct 30 06:22:36.566580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:36.566364 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" containerID="cri-o://047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091" gracePeriod=30 Oct 30 06:22:36.645950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:36.645912 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:36.969873 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:36.969830 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg"] Oct 30 06:22:36.970120 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:36.970090 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" containerID="cri-o://bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed" gracePeriod=30 Oct 30 06:22:37.368414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:37.368332 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:37.369501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:37.369477 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:22:41.363502 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.363460 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:22:41.363906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.363537 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:22:41.859527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.859493 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:22:41.975167 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975134 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-token\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-token\") pod \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " Oct 30 06:22:41.975389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975187 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istiod-ca-cert\") pod \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " Oct 30 06:22:41.975389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975230 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-data\") pod \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " Oct 30 06:22:41.975389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975272 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-certs\") pod \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " Oct 30 06:22:41.975556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975403 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-kube-api-access-ts9kn\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-kube-api-access-ts9kn\") pod \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " Oct 30 06:22:41.975556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975432 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-podinfo\") pod \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " Oct 30 06:22:41.975670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975564 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-credential-socket\") pod \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " Oct 30 06:22:41.975670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975652 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-envoy\") pod \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " Oct 30 06:22:41.975775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975714 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-socket\") pod \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\" (UID: \"a6f273d9-6d40-407b-ab96-0da3ea0f3582\") " Oct 30 06:22:41.975912 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.975858 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-credential-socket" pod "a6f273d9-6d40-407b-ab96-0da3ea0f3582" (UID: "a6f273d9-6d40-407b-ab96-0da3ea0f3582"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:41.976083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.976027 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-certs" pod "a6f273d9-6d40-407b-ab96-0da3ea0f3582" (UID: "a6f273d9-6d40-407b-ab96-0da3ea0f3582"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:41.976191 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.976160 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-socket" pod "a6f273d9-6d40-407b-ab96-0da3ea0f3582" (UID: "a6f273d9-6d40-407b-ab96-0da3ea0f3582"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:41.976361 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.976340 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istiod-ca-cert" pod "a6f273d9-6d40-407b-ab96-0da3ea0f3582" (UID: "a6f273d9-6d40-407b-ab96-0da3ea0f3582"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:41.976448 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.976381 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:41.976448 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.976398 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:41.976448 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.976418 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:41.976625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.976467 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-data" pod "a6f273d9-6d40-407b-ab96-0da3ea0f3582" (UID: "a6f273d9-6d40-407b-ab96-0da3ea0f3582"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:41.978195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.978171 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-token" pod "a6f273d9-6d40-407b-ab96-0da3ea0f3582" (UID: "a6f273d9-6d40-407b-ab96-0da3ea0f3582"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:41.978765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.978738 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-podinfo" pod "a6f273d9-6d40-407b-ab96-0da3ea0f3582" (UID: "a6f273d9-6d40-407b-ab96-0da3ea0f3582"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:41.978868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.978826 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-kube-api-access-ts9kn" pod "a6f273d9-6d40-407b-ab96-0da3ea0f3582" (UID: "a6f273d9-6d40-407b-ab96-0da3ea0f3582"). InnerVolumeSpecName "kube-api-access-ts9kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:41.978868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:41.978861 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-envoy" pod "a6f273d9-6d40-407b-ab96-0da3ea0f3582" (UID: "a6f273d9-6d40-407b-ab96-0da3ea0f3582"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:42.077807 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.077768 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts9kn\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-kube-api-access-ts9kn\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.077807 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.077797 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.077807 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.077807 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.077807 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.077816 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.078202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.077826 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.078202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.077836 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f273d9-6d40-407b-ab96-0da3ea0f3582-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.229927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.229866 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:22:42.387401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387356 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-socket\") pod \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " Oct 30 06:22:42.387942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387441 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-kube-api-access-zj642\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-kube-api-access-zj642\") pod \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " Oct 30 06:22:42.387942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387469 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-podinfo\") pod \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " Oct 30 06:22:42.387942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387525 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-data\") pod \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " Oct 30 06:22:42.387942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387668 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-socket" pod "16757c77-6f4b-4447-abcc-4c5cbf3d27cc" (UID: "16757c77-6f4b-4447-abcc-4c5cbf3d27cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:42.387942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387686 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-token\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-token\") pod \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " Oct 30 06:22:42.387942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387793 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-envoy\") pod \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " Oct 30 06:22:42.387942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387918 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-credential-socket\") pod \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " Oct 30 06:22:42.387942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387918 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-data" pod "16757c77-6f4b-4447-abcc-4c5cbf3d27cc" (UID: "16757c77-6f4b-4447-abcc-4c5cbf3d27cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:42.388415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.387967 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istiod-ca-cert\") pod \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " Oct 30 06:22:42.388415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.388021 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-certs\") pod \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\" (UID: \"16757c77-6f4b-4447-abcc-4c5cbf3d27cc\") " Oct 30 06:22:42.388415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.388230 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-credential-socket" pod "16757c77-6f4b-4447-abcc-4c5cbf3d27cc" (UID: "16757c77-6f4b-4447-abcc-4c5cbf3d27cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:42.388581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.388511 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-certs" pod "16757c77-6f4b-4447-abcc-4c5cbf3d27cc" (UID: "16757c77-6f4b-4447-abcc-4c5cbf3d27cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:42.388629 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.388570 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istiod-ca-cert" pod "16757c77-6f4b-4447-abcc-4c5cbf3d27cc" (UID: "16757c77-6f4b-4447-abcc-4c5cbf3d27cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:42.388745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.388725 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.388791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.388752 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.388791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.388785 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.388864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.388801 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.388864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.388815 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.390281 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.390259 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-token" pod "16757c77-6f4b-4447-abcc-4c5cbf3d27cc" (UID: "16757c77-6f4b-4447-abcc-4c5cbf3d27cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:42.390402 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.390272 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-podinfo" pod "16757c77-6f4b-4447-abcc-4c5cbf3d27cc" (UID: "16757c77-6f4b-4447-abcc-4c5cbf3d27cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:42.390402 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.390283 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-kube-api-access-zj642" pod "16757c77-6f4b-4447-abcc-4c5cbf3d27cc" (UID: "16757c77-6f4b-4447-abcc-4c5cbf3d27cc"). InnerVolumeSpecName "kube-api-access-zj642". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:42.390480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.390398 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-envoy" pod "16757c77-6f4b-4447-abcc-4c5cbf3d27cc" (UID: "16757c77-6f4b-4447-abcc-4c5cbf3d27cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:42.406769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.406730 4431 generic.go:356] "Generic (PLEG): container finished" podID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerID="047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091" exitCode=0 Oct 30 06:22:42.406938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.406798 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" Oct 30 06:22:42.406938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.406804 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" event={"ID":"a6f273d9-6d40-407b-ab96-0da3ea0f3582","Type":"ContainerDied","Data":"047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091"} Oct 30 06:22:42.406938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.406862 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh" event={"ID":"a6f273d9-6d40-407b-ab96-0da3ea0f3582","Type":"ContainerDied","Data":"1fe4906c862f5477a5153f6467a69022a313d4588496d2490a09df422129dbb5"} Oct 30 06:22:42.406938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.406886 4431 scope.go:122] "RemoveContainer" containerID="047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091" Oct 30 06:22:42.408333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.408272 4431 generic.go:356] "Generic (PLEG): container finished" podID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerID="bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed" exitCode=0 Oct 30 06:22:42.408459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.408362 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" Oct 30 06:22:42.408459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.408352 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" event={"ID":"16757c77-6f4b-4447-abcc-4c5cbf3d27cc","Type":"ContainerDied","Data":"bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed"} Oct 30 06:22:42.408557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.408484 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg" event={"ID":"16757c77-6f4b-4447-abcc-4c5cbf3d27cc","Type":"ContainerDied","Data":"00bc6a1c708ed2f2dc34d8333fdbd5448aab3070ef46e90a25afc50d9e7ced84"} Oct 30 06:22:42.419035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.418912 4431 scope.go:122] "RemoveContainer" containerID="047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091" Oct 30 06:22:42.419379 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:22:42.419351 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091\": container with ID starting with 047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091 not found: ID does not exist" containerID="047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091" Oct 30 06:22:42.419483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.419389 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091"} err="failed to get container status \"047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091\": rpc error: code = NotFound desc = could not find container \"047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091\": container with ID starting with 047fd327960030dec3271edbd438fbe17bec38585097c4c2006068ba700d4091 not found: ID does not exist" Oct 30 06:22:42.419483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.419416 4431 scope.go:122] "RemoveContainer" containerID="bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed" Oct 30 06:22:42.428635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.428613 4431 scope.go:122] "RemoveContainer" containerID="bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed" Oct 30 06:22:42.428927 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:22:42.428907 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed\": container with ID starting with bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed not found: ID does not exist" containerID="bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed" Oct 30 06:22:42.429002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.428934 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed"} err="failed to get container status \"bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed\": rpc error: code = NotFound desc = could not find container \"bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed\": container with ID starting with bc0e8a0a765e9de4ae91a2e1643b68ae7777148367c6988ffbcae39e671dcaed not found: ID does not exist" Oct 30 06:22:42.465348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.465289 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg"] Oct 30 06:22:42.468619 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.468584 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-vtxpg"] Oct 30 06:22:42.490337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.490235 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.490337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.490271 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.490337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.490285 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj642\" (UniqueName: \"kubernetes.io/projected/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-kube-api-access-zj642\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.490337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.490312 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16757c77-6f4b-4447-abcc-4c5cbf3d27cc-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:22:42.504040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.504002 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh"] Oct 30 06:22:42.508130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:42.508096 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cjmgh"] Oct 30 06:22:43.337940 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:43.337903 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" path="/var/lib/kubelet/pods/16757c77-6f4b-4447-abcc-4c5cbf3d27cc/volumes" Oct 30 06:22:43.338465 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:43.338450 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" path="/var/lib/kubelet/pods/a6f273d9-6d40-407b-ab96-0da3ea0f3582/volumes" Oct 30 06:22:46.298922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:46.298890 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hpphs_9b2cce36-89ca-4ea5-a488-be5af53dbf14/payload-processing/0.log" Oct 30 06:22:46.514273 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:46.514241 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hpphs_9b2cce36-89ca-4ea5-a488-be5af53dbf14/payload-processing/0.log" Oct 30 06:22:58.055618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:58.055575 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w"] Oct 30 06:22:58.056109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:22:58.055821 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" containerID="cri-o://b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80" gracePeriod=30 Oct 30 06:23:03.339889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.339859 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:23:03.455127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455068 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-certs\") pod \"2abf654b-4fe7-448e-a7ba-05be6478a21d\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " Oct 30 06:23:03.455381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455149 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-credential-socket\") pod \"2abf654b-4fe7-448e-a7ba-05be6478a21d\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " Oct 30 06:23:03.455381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455194 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-podinfo\") pod \"2abf654b-4fe7-448e-a7ba-05be6478a21d\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " Oct 30 06:23:03.455381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455272 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-envoy\") pod \"2abf654b-4fe7-448e-a7ba-05be6478a21d\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " Oct 30 06:23:03.455381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455368 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-socket\") pod \"2abf654b-4fe7-448e-a7ba-05be6478a21d\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " Oct 30 06:23:03.455620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455403 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-token\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-token\") pod \"2abf654b-4fe7-448e-a7ba-05be6478a21d\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " Oct 30 06:23:03.455620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455419 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-certs" pod "2abf654b-4fe7-448e-a7ba-05be6478a21d" (UID: "2abf654b-4fe7-448e-a7ba-05be6478a21d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.455620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455507 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-kube-api-access-8zwrs\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-kube-api-access-8zwrs\") pod \"2abf654b-4fe7-448e-a7ba-05be6478a21d\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " Oct 30 06:23:03.455620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455514 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-credential-socket" pod "2abf654b-4fe7-448e-a7ba-05be6478a21d" (UID: "2abf654b-4fe7-448e-a7ba-05be6478a21d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.455620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455553 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2abf654b-4fe7-448e-a7ba-05be6478a21d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf654b-4fe7-448e-a7ba-05be6478a21d-istiod-ca-cert\") pod \"2abf654b-4fe7-448e-a7ba-05be6478a21d\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " Oct 30 06:23:03.455620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455604 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-data\") pod \"2abf654b-4fe7-448e-a7ba-05be6478a21d\" (UID: \"2abf654b-4fe7-448e-a7ba-05be6478a21d\") " Oct 30 06:23:03.455904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455665 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-socket" pod "2abf654b-4fe7-448e-a7ba-05be6478a21d" (UID: "2abf654b-4fe7-448e-a7ba-05be6478a21d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.455973 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455941 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abf654b-4fe7-448e-a7ba-05be6478a21d-istiod-ca-cert" pod "2abf654b-4fe7-448e-a7ba-05be6478a21d" (UID: "2abf654b-4fe7-448e-a7ba-05be6478a21d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:03.455973 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.455936 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-data" pod "2abf654b-4fe7-448e-a7ba-05be6478a21d" (UID: "2abf654b-4fe7-448e-a7ba-05be6478a21d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.456263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.456243 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.456349 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.456269 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.456349 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.456286 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.456349 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.456334 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf654b-4fe7-448e-a7ba-05be6478a21d-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.456349 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.456348 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.457766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.457739 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-envoy" pod "2abf654b-4fe7-448e-a7ba-05be6478a21d" (UID: "2abf654b-4fe7-448e-a7ba-05be6478a21d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.457856 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.457749 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-kube-api-access-8zwrs" pod "2abf654b-4fe7-448e-a7ba-05be6478a21d" (UID: "2abf654b-4fe7-448e-a7ba-05be6478a21d"). InnerVolumeSpecName "kube-api-access-8zwrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:03.457990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.457968 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-token" pod "2abf654b-4fe7-448e-a7ba-05be6478a21d" (UID: "2abf654b-4fe7-448e-a7ba-05be6478a21d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:03.457990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.457982 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-podinfo" pod "2abf654b-4fe7-448e-a7ba-05be6478a21d" (UID: "2abf654b-4fe7-448e-a7ba-05be6478a21d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:03.535319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.535267 4431 generic.go:356] "Generic (PLEG): container finished" podID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerID="b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80" exitCode=0 Oct 30 06:23:03.535499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.535340 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" Oct 30 06:23:03.535499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.535344 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" event={"ID":"2abf654b-4fe7-448e-a7ba-05be6478a21d","Type":"ContainerDied","Data":"b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80"} Oct 30 06:23:03.535499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.535419 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w" event={"ID":"2abf654b-4fe7-448e-a7ba-05be6478a21d","Type":"ContainerDied","Data":"b1c02c7b0854a0c8bfd4fd3e3ed8c7a1769d8d5050babc02374ca1107bf023f0"} Oct 30 06:23:03.535499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.535451 4431 scope.go:122] "RemoveContainer" containerID="b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80" Oct 30 06:23:03.546129 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.546111 4431 scope.go:122] "RemoveContainer" containerID="b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80" Oct 30 06:23:03.546498 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:23:03.546477 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80\": container with ID starting with b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80 not found: ID does not exist" containerID="b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80" Oct 30 06:23:03.546569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.546508 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80"} err="failed to get container status \"b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80\": rpc error: code = NotFound desc = could not find container \"b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80\": container with ID starting with b43849a8f4fd7db880d8b446d757a662f49d917da78e54a13f80cb8d70117d80 not found: ID does not exist" Oct 30 06:23:03.557899 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.557873 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.558001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.557902 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.558001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.557915 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.558001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.557923 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zwrs\" (UniqueName: \"kubernetes.io/projected/2abf654b-4fe7-448e-a7ba-05be6478a21d-kube-api-access-8zwrs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.588293 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.588199 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w"] Oct 30 06:23:03.592442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:03.592409 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2q89w"] Oct 30 06:23:03.702970 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:23:03.702928 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:23:05.337935 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:05.337889 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" path="/var/lib/kubelet/pods/2abf654b-4fe7-448e-a7ba-05be6478a21d/volumes" Oct 30 06:23:32.150078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:32.150042 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4"] Oct 30 06:23:32.151650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:32.151609 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" podUID="ab1a329d-b0a9-459b-9dbb-8cb595aa640c" containerName="maas-api" containerID="cri-o://0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8" gracePeriod=30 Oct 30 06:23:33.514642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.514614 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:23:33.693020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.692975 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-openshift-service-ca\") pod \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " Oct 30 06:23:33.693273 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.693123 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-kube-api-access-x7vnk\" (UniqueName: \"kubernetes.io/projected/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-kube-api-access-x7vnk\") pod \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " Oct 30 06:23:33.693273 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.693170 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-maas-api-tls\") pod \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " Oct 30 06:23:33.693499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.693334 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-metrics-certs\") pod \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\" (UID: \"ab1a329d-b0a9-459b-9dbb-8cb595aa640c\") " Oct 30 06:23:33.695491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.695443 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-openshift-service-ca" pod "ab1a329d-b0a9-459b-9dbb-8cb595aa640c" (UID: "ab1a329d-b0a9-459b-9dbb-8cb595aa640c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:33.726557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.715452 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-maas-api-tls" pod "ab1a329d-b0a9-459b-9dbb-8cb595aa640c" (UID: "ab1a329d-b0a9-459b-9dbb-8cb595aa640c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:33.732469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.730665 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-kube-api-access-x7vnk" pod "ab1a329d-b0a9-459b-9dbb-8cb595aa640c" (UID: "ab1a329d-b0a9-459b-9dbb-8cb595aa640c"). InnerVolumeSpecName "kube-api-access-x7vnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:33.741006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.740840 4431 generic.go:356] "Generic (PLEG): container finished" podID="ab1a329d-b0a9-459b-9dbb-8cb595aa640c" containerID="0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8" exitCode=0 Oct 30 06:23:33.741483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.741426 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" event={"ID":"ab1a329d-b0a9-459b-9dbb-8cb595aa640c","Type":"ContainerDied","Data":"0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8"} Oct 30 06:23:33.741727 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.741681 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" event={"ID":"ab1a329d-b0a9-459b-9dbb-8cb595aa640c","Type":"ContainerDied","Data":"89a9d9e78dfe76dd6212ca25b851bf995d5b84f01ce408091dc43a16aa8e74f4"} Oct 30 06:23:33.742685 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.741901 4431 scope.go:122] "RemoveContainer" containerID="0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8" Oct 30 06:23:33.742685 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.742101 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4" Oct 30 06:23:33.748958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.746453 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-metrics-certs" pod "ab1a329d-b0a9-459b-9dbb-8cb595aa640c" (UID: "ab1a329d-b0a9-459b-9dbb-8cb595aa640c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:33.791923 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.791537 4431 scope.go:122] "RemoveContainer" containerID="0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8" Oct 30 06:23:33.792552 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:23:33.792515 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8\": container with ID starting with 0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8 not found: ID does not exist" containerID="0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8" Oct 30 06:23:33.792688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.792565 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8"} err="failed to get container status \"0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8\": rpc error: code = NotFound desc = could not find container \"0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8\": container with ID starting with 0f84998cbc76237066062c49b992bcf2b4995a647f12a1e64051e58ebe505ba8 not found: ID does not exist" Oct 30 06:23:33.795704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.795678 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7vnk\" (UniqueName: \"kubernetes.io/projected/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-kube-api-access-x7vnk\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:33.795704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.795704 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:33.795859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.795713 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:33.795859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:33.795723 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a329d-b0a9-459b-9dbb-8cb595aa640c-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:23:33.797617 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:23:33.797579 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:23:34.096097 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:34.096053 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4"] Oct 30 06:23:34.108090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:34.108044 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-532f3b-d98798bdd-6f6p4"] Oct 30 06:23:35.338848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:35.338800 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1a329d-b0a9-459b-9dbb-8cb595aa640c" path="/var/lib/kubelet/pods/ab1a329d-b0a9-459b-9dbb-8cb595aa640c/volumes" Oct 30 06:23:42.831368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:42.831292 4431 generic.go:356] "Generic (PLEG): container finished" podID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerID="2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17" exitCode=0 Oct 30 06:23:42.831940 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:42.831374 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" event={"ID":"a7ab613a-ebe8-40c9-88f6-559d74969539","Type":"ContainerDied","Data":"2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17"} Oct 30 06:23:42.831940 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:42.831882 4431 scope.go:122] "RemoveContainer" containerID="2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17" Oct 30 06:23:42.833634 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:42.833608 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:23:42.833721 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:42.833683 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:23:42.833721 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:42.833714 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:23:43.839057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:43.839022 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" event={"ID":"a7ab613a-ebe8-40c9-88f6-559d74969539","Type":"ContainerStarted","Data":"400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a"} Oct 30 06:23:44.638649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:44.638606 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:23:44.638887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:44.638858 4431 patch_prober.go:29] interesting pod/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 06:23:44.638978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:44.638943 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 06:23:45.639479 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:45.639441 4431 patch_prober.go:29] interesting pod/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 06:23:45.639982 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:45.639509 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 06:23:46.638983 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:46.638952 4431 patch_prober.go:29] interesting pod/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 06:23:46.639181 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:46.639007 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 06:23:47.639090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:47.639055 4431 patch_prober.go:29] interesting pod/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 06:23:47.639547 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:47.639114 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 06:23:48.638659 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:48.638625 4431 patch_prober.go:29] interesting pod/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 06:23:48.638850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:48.638707 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 06:23:49.644010 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:49.643980 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:23:49.644478 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:49.644251 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:23:49.645124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:49.645105 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:23:57.970315 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:57.970218 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w"] Oct 30 06:23:57.970781 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:57.970442 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" containerID="cri-o://33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0" gracePeriod=30 Oct 30 06:23:57.983465 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:57.983430 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh"] Oct 30 06:23:57.983669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:23:57.983642 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" containerID="cri-o://8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e" gracePeriod=30 Oct 30 06:24:00.019617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.019580 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl"] Oct 30 06:24:00.020100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020071 4431 cpu_manager.go:414] "containerMap: removing container" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" Oct 30 06:24:00.020100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020083 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" Oct 30 06:24:00.020213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020106 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ab1a329d-b0a9-459b-9dbb-8cb595aa640c" containerName="maas-api" Oct 30 06:24:00.020213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020111 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1a329d-b0a9-459b-9dbb-8cb595aa640c" containerName="maas-api" Oct 30 06:24:00.020213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020119 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" Oct 30 06:24:00.020213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020123 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" Oct 30 06:24:00.020213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020145 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" Oct 30 06:24:00.020213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020149 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" Oct 30 06:24:00.020550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020220 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1a329d-b0a9-459b-9dbb-8cb595aa640c" containerName="maas-api" Oct 30 06:24:00.020550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020229 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abf654b-4fe7-448e-a7ba-05be6478a21d" containerName="istio-proxy" Oct 30 06:24:00.020550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020238 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f273d9-6d40-407b-ab96-0da3ea0f3582" containerName="istio-proxy" Oct 30 06:24:00.020550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.020245 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="16757c77-6f4b-4447-abcc-4c5cbf3d27cc" containerName="istio-proxy" Oct 30 06:24:00.022995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.022973 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.041950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.041927 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-532f3b-openshift-default-dockercfg-hfslq\"" Oct 30 06:24:00.045605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.045577 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92926866-72a3-4f6a-922c-1ae2a98c55d7-istiod-ca-cert\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.045720 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.045651 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-certs\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.045774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.045763 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.045811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.045794 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-data\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.045913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.045892 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-credential-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.046041 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.046017 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6bb\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-kube-api-access-rn6bb\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.046107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.046071 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-envoy\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.046214 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.046176 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-podinfo\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.046426 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.046403 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-token\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.057110 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.057082 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl"] Oct 30 06:24:00.147098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147071 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-credential-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147145 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6bb\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-kube-api-access-rn6bb\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147178 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-envoy\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147220 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-podinfo\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147258 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-token\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147342 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92926866-72a3-4f6a-922c-1ae2a98c55d7-istiod-ca-cert\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147395 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-certs\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147427 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147455 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-data\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147516 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-credential-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147754 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-certs\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147803 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-data\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.147843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.147831 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-socket\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.148166 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.148140 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92926866-72a3-4f6a-922c-1ae2a98c55d7-istiod-ca-cert\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.149548 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.149523 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-envoy\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.149869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.149846 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-podinfo\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.170974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.170950 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-token\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.171080 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.171065 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6bb\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-kube-api-access-rn6bb\") pod \"e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.335591 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.335508 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:00.664721 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:24:00.664692 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92926866_72a3_4f6a_922c_1ae2a98c55d7.slice/crio-f9e428f47d97915367dbd2564e10121d338112d7dfe2bcfaae23f301448c8119 WatchSource:0}: Error finding container f9e428f47d97915367dbd2564e10121d338112d7dfe2bcfaae23f301448c8119: Status 404 returned error can't find the container with id f9e428f47d97915367dbd2564e10121d338112d7dfe2bcfaae23f301448c8119 Oct 30 06:24:00.665023 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.665001 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl"] Oct 30 06:24:00.667331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.667284 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:24:00.667401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.667367 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:24:00.667401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.667395 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:24:00.936643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.936552 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" event={"ID":"92926866-72a3-4f6a-922c-1ae2a98c55d7","Type":"ContainerStarted","Data":"9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33"} Oct 30 06:24:00.936643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.936625 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" event={"ID":"92926866-72a3-4f6a-922c-1ae2a98c55d7","Type":"ContainerStarted","Data":"f9e428f47d97915367dbd2564e10121d338112d7dfe2bcfaae23f301448c8119"} Oct 30 06:24:00.936998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.936972 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:24:00.937136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.937025 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:24:00.987321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:00.983884 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" podStartSLOduration=1.983866923 podStartE2EDuration="1.983866923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:00.982182883 +0000 UTC m=+1849.788704636" watchObservedRunningTime="2026-10-30 06:24:00.983866923 +0000 UTC m=+1849.790388673" Oct 30 06:24:01.337889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:01.337852 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:01.341929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:01.341906 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:01.942593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:01.942557 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:01.943556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:01.943527 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:03.264924 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.264893 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:24:03.285979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.285946 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-certs\") pod \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " Oct 30 06:24:03.286147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286029 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istiod-ca-cert\") pod \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " Oct 30 06:24:03.286147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286106 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-envoy\") pod \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " Oct 30 06:24:03.286147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286133 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-podinfo\") pod \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " Oct 30 06:24:03.286364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286190 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-socket\") pod \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " Oct 30 06:24:03.286364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286224 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-token\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-token\") pod \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " Oct 30 06:24:03.286364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286240 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-certs" pod "84beb3ad-bff6-4552-b721-dc7a05b4cc92" (UID: "84beb3ad-bff6-4552-b721-dc7a05b4cc92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.286364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286251 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-credential-socket\") pod \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " Oct 30 06:24:03.286560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286403 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-data\") pod \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " Oct 30 06:24:03.286560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286463 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-kube-api-access-gsvkh\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-kube-api-access-gsvkh\") pod \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\" (UID: \"84beb3ad-bff6-4552-b721-dc7a05b4cc92\") " Oct 30 06:24:03.286560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286489 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-credential-socket" pod "84beb3ad-bff6-4552-b721-dc7a05b4cc92" (UID: "84beb3ad-bff6-4552-b721-dc7a05b4cc92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.286690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286508 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istiod-ca-cert" pod "84beb3ad-bff6-4552-b721-dc7a05b4cc92" (UID: "84beb3ad-bff6-4552-b721-dc7a05b4cc92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:03.287390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.286856 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-data" pod "84beb3ad-bff6-4552-b721-dc7a05b4cc92" (UID: "84beb3ad-bff6-4552-b721-dc7a05b4cc92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.287390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.287131 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.287390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.287155 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.287390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.287170 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.287390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.287188 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.287390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.287272 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-socket" pod "84beb3ad-bff6-4552-b721-dc7a05b4cc92" (UID: "84beb3ad-bff6-4552-b721-dc7a05b4cc92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.288631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.288602 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-envoy" pod "84beb3ad-bff6-4552-b721-dc7a05b4cc92" (UID: "84beb3ad-bff6-4552-b721-dc7a05b4cc92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.288784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.288758 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-token" pod "84beb3ad-bff6-4552-b721-dc7a05b4cc92" (UID: "84beb3ad-bff6-4552-b721-dc7a05b4cc92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:03.288828 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.288789 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-kube-api-access-gsvkh" pod "84beb3ad-bff6-4552-b721-dc7a05b4cc92" (UID: "84beb3ad-bff6-4552-b721-dc7a05b4cc92"). InnerVolumeSpecName "kube-api-access-gsvkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:03.289397 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.289382 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-podinfo" pod "84beb3ad-bff6-4552-b721-dc7a05b4cc92" (UID: "84beb3ad-bff6-4552-b721-dc7a05b4cc92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:03.290318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.290287 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:24:03.387672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.387636 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istiod-ca-cert\") pod \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " Oct 30 06:24:03.387871 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.387720 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-envoy\") pod \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " Oct 30 06:24:03.387871 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.387769 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-token\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-token\") pod \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " Oct 30 06:24:03.387992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.387868 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-credential-socket\") pod \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " Oct 30 06:24:03.387992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.387903 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-podinfo\") pod \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " Oct 30 06:24:03.388091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.388009 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-kube-api-access-8ht8t\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-kube-api-access-8ht8t\") pod \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " Oct 30 06:24:03.388091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.388042 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istiod-ca-cert" pod "4e444187-d3e6-4f34-9a68-42eb1c6f7260" (UID: "4e444187-d3e6-4f34-9a68-42eb1c6f7260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:03.388091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.388062 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-certs\") pod \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " Oct 30 06:24:03.388251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.388138 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-data\") pod \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " Oct 30 06:24:03.388251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.388166 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-socket\") pod \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\" (UID: \"4e444187-d3e6-4f34-9a68-42eb1c6f7260\") " Oct 30 06:24:03.388251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.388204 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-credential-socket" pod "4e444187-d3e6-4f34-9a68-42eb1c6f7260" (UID: "4e444187-d3e6-4f34-9a68-42eb1c6f7260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.388484 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.388460 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-certs" pod "4e444187-d3e6-4f34-9a68-42eb1c6f7260" (UID: "4e444187-d3e6-4f34-9a68-42eb1c6f7260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.388955 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.388924 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-data" pod "4e444187-d3e6-4f34-9a68-42eb1c6f7260" (UID: "4e444187-d3e6-4f34-9a68-42eb1c6f7260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.389104 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.389080 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.389168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.389112 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.389168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.389126 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.389168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.389144 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84beb3ad-bff6-4552-b721-dc7a05b4cc92-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.389168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.389158 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.389449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.389171 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.389449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.389186 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsvkh\" (UniqueName: \"kubernetes.io/projected/84beb3ad-bff6-4552-b721-dc7a05b4cc92-kube-api-access-gsvkh\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.389449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.389200 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.389449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.389216 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.390144 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.390110 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-socket" pod "4e444187-d3e6-4f34-9a68-42eb1c6f7260" (UID: "4e444187-d3e6-4f34-9a68-42eb1c6f7260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.390981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.390953 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-envoy" pod "4e444187-d3e6-4f34-9a68-42eb1c6f7260" (UID: "4e444187-d3e6-4f34-9a68-42eb1c6f7260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:03.391235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.391210 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-podinfo" pod "4e444187-d3e6-4f34-9a68-42eb1c6f7260" (UID: "4e444187-d3e6-4f34-9a68-42eb1c6f7260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:03.391592 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.391569 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-kube-api-access-8ht8t" pod "4e444187-d3e6-4f34-9a68-42eb1c6f7260" (UID: "4e444187-d3e6-4f34-9a68-42eb1c6f7260"). InnerVolumeSpecName "kube-api-access-8ht8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:03.391870 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.391853 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-token" pod "4e444187-d3e6-4f34-9a68-42eb1c6f7260" (UID: "4e444187-d3e6-4f34-9a68-42eb1c6f7260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:03.490581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.490490 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.490581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.490519 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.490581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.490528 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.490581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.490536 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e444187-d3e6-4f34-9a68-42eb1c6f7260-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.490581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.490548 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ht8t\" (UniqueName: \"kubernetes.io/projected/4e444187-d3e6-4f34-9a68-42eb1c6f7260-kube-api-access-8ht8t\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:03.703484 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:24:03.703448 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:24:03.967121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.967079 4431 generic.go:356] "Generic (PLEG): container finished" podID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerID="8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e" exitCode=0 Oct 30 06:24:03.967365 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.967134 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" Oct 30 06:24:03.967365 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.967130 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" event={"ID":"4e444187-d3e6-4f34-9a68-42eb1c6f7260","Type":"ContainerDied","Data":"8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e"} Oct 30 06:24:03.967365 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.967265 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh" event={"ID":"4e444187-d3e6-4f34-9a68-42eb1c6f7260","Type":"ContainerDied","Data":"498ba6b2fe6bad246a46c140e1c2b13e663beee1598562cdcf3b6f1ad880cc44"} Oct 30 06:24:03.967365 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.967296 4431 scope.go:122] "RemoveContainer" containerID="8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e" Oct 30 06:24:03.969277 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.969252 4431 generic.go:356] "Generic (PLEG): container finished" podID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerID="33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0" exitCode=0 Oct 30 06:24:03.969482 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.969453 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" Oct 30 06:24:03.969618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.969456 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" event={"ID":"84beb3ad-bff6-4552-b721-dc7a05b4cc92","Type":"ContainerDied","Data":"33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0"} Oct 30 06:24:03.969618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.969604 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w" event={"ID":"84beb3ad-bff6-4552-b721-dc7a05b4cc92","Type":"ContainerDied","Data":"7f33bf29b9df19f64c291d7087470d0a44cdb91fffa576537bc4a3e2f0bfb817"} Oct 30 06:24:03.977937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.977920 4431 scope.go:122] "RemoveContainer" containerID="8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e" Oct 30 06:24:03.978241 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:24:03.978223 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e\": container with ID starting with 8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e not found: ID does not exist" containerID="8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e" Oct 30 06:24:03.978282 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.978250 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e"} err="failed to get container status \"8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e\": rpc error: code = NotFound desc = could not find container \"8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e\": container with ID starting with 8317e5693b71472e528e363eece09c1400e3aaaaab3bb09a59ed70501820665e not found: ID does not exist" Oct 30 06:24:03.978282 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.978268 4431 scope.go:122] "RemoveContainer" containerID="33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0" Oct 30 06:24:03.987440 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:03.987419 4431 scope.go:122] "RemoveContainer" containerID="a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48" Oct 30 06:24:04.006840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:04.006819 4431 scope.go:122] "RemoveContainer" containerID="33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0" Oct 30 06:24:04.007181 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:24:04.007155 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0\": container with ID starting with 33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0 not found: ID does not exist" containerID="33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0" Oct 30 06:24:04.007255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:04.007189 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0"} err="failed to get container status \"33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0\": rpc error: code = NotFound desc = could not find container \"33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0\": container with ID starting with 33b2003846bc2115f5722ad4ce7fbdfcec54da5e434ffcb1ed2503e0c19fb8b0 not found: ID does not exist" Oct 30 06:24:04.007255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:04.007215 4431 scope.go:122] "RemoveContainer" containerID="a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48" Oct 30 06:24:04.007570 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:24:04.007542 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48\": container with ID starting with a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48 not found: ID does not exist" containerID="a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48" Oct 30 06:24:04.007662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:04.007578 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48"} err="failed to get container status \"a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48\": rpc error: code = NotFound desc = could not find container \"a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48\": container with ID starting with a825177545e63a0e9ae6eae1f2d84ad7f7fa69950920baf5613d1ff016141f48 not found: ID does not exist" Oct 30 06:24:04.020625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:04.020588 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w"] Oct 30 06:24:04.032404 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:04.032375 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-7694w"] Oct 30 06:24:04.062672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:04.062639 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh"] Oct 30 06:24:04.065127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:04.065095 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-z5fxh"] Oct 30 06:24:05.340145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:05.340101 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" path="/var/lib/kubelet/pods/4e444187-d3e6-4f34-9a68-42eb1c6f7260/volumes" Oct 30 06:24:05.341011 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:05.340982 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" path="/var/lib/kubelet/pods/84beb3ad-bff6-4552-b721-dc7a05b4cc92/volumes" Oct 30 06:24:24.444342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:24.444291 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl"] Oct 30 06:24:24.444974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:24.444922 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" podUID="92926866-72a3-4f6a-922c-1ae2a98c55d7" containerName="istio-proxy" containerID="cri-o://9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33" gracePeriod=30 Oct 30 06:24:29.727156 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.727127 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:29.834979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.834938 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-socket\") pod \"92926866-72a3-4f6a-922c-1ae2a98c55d7\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " Oct 30 06:24:29.835161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835034 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-certs\") pod \"92926866-72a3-4f6a-922c-1ae2a98c55d7\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " Oct 30 06:24:29.835161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835104 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-podinfo\") pod \"92926866-72a3-4f6a-922c-1ae2a98c55d7\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " Oct 30 06:24:29.835161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835150 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-kube-api-access-rn6bb\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-kube-api-access-rn6bb\") pod \"92926866-72a3-4f6a-922c-1ae2a98c55d7\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " Oct 30 06:24:29.835358 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835245 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-socket" pod "92926866-72a3-4f6a-922c-1ae2a98c55d7" (UID: "92926866-72a3-4f6a-922c-1ae2a98c55d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.835358 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835281 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-data\") pod \"92926866-72a3-4f6a-922c-1ae2a98c55d7\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " Oct 30 06:24:29.835358 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835348 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-token\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-token\") pod \"92926866-72a3-4f6a-922c-1ae2a98c55d7\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " Oct 30 06:24:29.835513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835384 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-certs" pod "92926866-72a3-4f6a-922c-1ae2a98c55d7" (UID: "92926866-72a3-4f6a-922c-1ae2a98c55d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.835513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835417 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-envoy\") pod \"92926866-72a3-4f6a-922c-1ae2a98c55d7\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " Oct 30 06:24:29.835632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835522 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-credential-socket\") pod \"92926866-72a3-4f6a-922c-1ae2a98c55d7\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " Oct 30 06:24:29.835632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835567 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-data" pod "92926866-72a3-4f6a-922c-1ae2a98c55d7" (UID: "92926866-72a3-4f6a-922c-1ae2a98c55d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.835814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835790 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-credential-socket" pod "92926866-72a3-4f6a-922c-1ae2a98c55d7" (UID: "92926866-72a3-4f6a-922c-1ae2a98c55d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.835879 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835817 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92926866-72a3-4f6a-922c-1ae2a98c55d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92926866-72a3-4f6a-922c-1ae2a98c55d7-istiod-ca-cert\") pod \"92926866-72a3-4f6a-922c-1ae2a98c55d7\" (UID: \"92926866-72a3-4f6a-922c-1ae2a98c55d7\") " Oct 30 06:24:29.836070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.835967 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92926866-72a3-4f6a-922c-1ae2a98c55d7-istiod-ca-cert" pod "92926866-72a3-4f6a-922c-1ae2a98c55d7" (UID: "92926866-72a3-4f6a-922c-1ae2a98c55d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:29.836440 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.836419 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.836541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.836447 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92926866-72a3-4f6a-922c-1ae2a98c55d7-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.836541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.836464 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.836541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.836479 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.836541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.836493 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.837663 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.837638 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-token" pod "92926866-72a3-4f6a-922c-1ae2a98c55d7" (UID: "92926866-72a3-4f6a-922c-1ae2a98c55d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:29.837771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.837665 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-kube-api-access-rn6bb" pod "92926866-72a3-4f6a-922c-1ae2a98c55d7" (UID: "92926866-72a3-4f6a-922c-1ae2a98c55d7"). InnerVolumeSpecName "kube-api-access-rn6bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:29.837827 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.837768 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-podinfo" pod "92926866-72a3-4f6a-922c-1ae2a98c55d7" (UID: "92926866-72a3-4f6a-922c-1ae2a98c55d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:29.837929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.837912 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-envoy" pod "92926866-72a3-4f6a-922c-1ae2a98c55d7" (UID: "92926866-72a3-4f6a-922c-1ae2a98c55d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:29.937790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.937754 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.937790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.937784 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn6bb\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-kube-api-access-rn6bb\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.937790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.937794 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.938052 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:29.937808 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92926866-72a3-4f6a-922c-1ae2a98c55d7-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:30.138801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:30.138700 4431 generic.go:356] "Generic (PLEG): container finished" podID="92926866-72a3-4f6a-922c-1ae2a98c55d7" containerID="9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33" exitCode=0 Oct 30 06:24:30.138801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:30.138762 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" Oct 30 06:24:30.138801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:30.138770 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" event={"ID":"92926866-72a3-4f6a-922c-1ae2a98c55d7","Type":"ContainerDied","Data":"9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33"} Oct 30 06:24:30.139067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:30.138826 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl" event={"ID":"92926866-72a3-4f6a-922c-1ae2a98c55d7","Type":"ContainerDied","Data":"f9e428f47d97915367dbd2564e10121d338112d7dfe2bcfaae23f301448c8119"} Oct 30 06:24:30.139067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:30.138857 4431 scope.go:122] "RemoveContainer" containerID="9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33" Oct 30 06:24:30.150723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:30.150700 4431 scope.go:122] "RemoveContainer" containerID="9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33" Oct 30 06:24:30.151038 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:24:30.151018 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33\": container with ID starting with 9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33 not found: ID does not exist" containerID="9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33" Oct 30 06:24:30.151080 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:30.151051 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33"} err="failed to get container status \"9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33\": rpc error: code = NotFound desc = could not find container \"9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33\": container with ID starting with 9ff0bc861406666dcd0e25d0f695b310a5eaaf52800109c4571a9a2a7b274c33 not found: ID does not exist" Oct 30 06:24:30.192006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:30.191951 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl"] Oct 30 06:24:30.204993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:30.204952 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-532f3b-openshift-default-5dcf76cb9c-tmbhl"] Oct 30 06:24:31.338816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:31.338782 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92926866-72a3-4f6a-922c-1ae2a98c55d7" path="/var/lib/kubelet/pods/92926866-72a3-4f6a-922c-1ae2a98c55d7/volumes" Oct 30 06:24:33.702026 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:24:33.701991 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:24:37.376153 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376113 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4"] Oct 30 06:24:37.376656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376641 4431 cpu_manager.go:414] "containerMap: removing container" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" Oct 30 06:24:37.376656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376656 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" Oct 30 06:24:37.376729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376678 4431 cpu_manager.go:414] "containerMap: removing container" podUID="92926866-72a3-4f6a-922c-1ae2a98c55d7" containerName="istio-proxy" Oct 30 06:24:37.376729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376684 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92926866-72a3-4f6a-922c-1ae2a98c55d7" containerName="istio-proxy" Oct 30 06:24:37.376729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376705 4431 cpu_manager.go:414] "containerMap: removing container" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" Oct 30 06:24:37.376729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376711 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" Oct 30 06:24:37.376850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376780 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" Oct 30 06:24:37.376850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376790 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="92926866-72a3-4f6a-922c-1ae2a98c55d7" containerName="istio-proxy" Oct 30 06:24:37.376850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376798 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" Oct 30 06:24:37.376850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.376805 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e444187-d3e6-4f34-9a68-42eb1c6f7260" containerName="istio-proxy" Oct 30 06:24:37.380210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.380191 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.413251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.413203 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4"] Oct 30 06:24:37.535127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.535077 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.535364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.535151 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.535364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.535246 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.535522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.535360 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.535522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.535439 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.535522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.535471 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs6gw\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-kube-api-access-gs6gw\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.535707 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.535531 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.535707 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.535573 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.535829 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.535709 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.637093 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.636998 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.637093 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637079 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.637371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637146 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.637371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637179 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs6gw\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-kube-api-access-gs6gw\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.637371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637204 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.637371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637231 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.637371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637286 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.638206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637375 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.638206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637437 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.638206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637605 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.638206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637914 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.638206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637944 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.638206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.637991 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.638206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.638174 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.640248 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.640224 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.640914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.640890 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.660702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.660661 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.660846 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.660786 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs6gw\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-kube-api-access-gs6gw\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:37.697801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:37.697739 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:38.017422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:38.017388 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4"] Oct 30 06:24:38.017703 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:24:38.017674 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131ee3df_7381_4d80_9ac5_5b0c104bcc01.slice/crio-90417fcf120b0aad87a1f54d5aa7a20e99ceb31d445c9408a9e85f0427116dc2 WatchSource:0}: Error finding container 90417fcf120b0aad87a1f54d5aa7a20e99ceb31d445c9408a9e85f0427116dc2: Status 404 returned error can't find the container with id 90417fcf120b0aad87a1f54d5aa7a20e99ceb31d445c9408a9e85f0427116dc2 Oct 30 06:24:38.020704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:38.020661 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:24:38.020826 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:38.020749 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:24:38.020826 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:38.020792 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:24:38.189545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:38.189508 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" event={"ID":"131ee3df-7381-4d80-9ac5-5b0c104bcc01","Type":"ContainerStarted","Data":"2887e42ff66885f6221dd218cdfc91cd48924ef11902fd12b36db7f8d961c61f"} Oct 30 06:24:38.189744 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:38.189568 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" event={"ID":"131ee3df-7381-4d80-9ac5-5b0c104bcc01","Type":"ContainerStarted","Data":"90417fcf120b0aad87a1f54d5aa7a20e99ceb31d445c9408a9e85f0427116dc2"} Oct 30 06:24:38.234663 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:38.234596 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" podStartSLOduration=1.234579522 podStartE2EDuration="1.234579522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:38.234085706 +0000 UTC m=+1887.040607492" watchObservedRunningTime="2026-10-30 06:24:38.234579522 +0000 UTC m=+1887.041101270" Oct 30 06:24:38.698209 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:38.698166 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:39.699714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:39.699680 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 06:24:39.700144 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:39.699733 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 06:24:40.698131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:40.698094 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 06:24:40.698332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:40.698147 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 06:24:41.698119 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:41.698083 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 06:24:41.698660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:41.698139 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 06:24:42.698872 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:42.698834 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 06:24:42.699373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:42.698898 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 06:24:43.698893 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:43.698847 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 06:24:43.699417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:43.698913 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 06:24:44.698687 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:44.698653 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 06:24:44.698881 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:44.698716 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 06:24:45.702866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:45.702830 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:45.703371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:45.703270 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:45.703921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:45.703899 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:24:47.924928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:47.924889 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f"] Oct 30 06:24:47.925512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:47.925147 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" podUID="0af70ae2-6fdb-455d-89f8-024f6003f52e" containerName="maas-api" containerID="cri-o://c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9" gracePeriod=30 Oct 30 06:24:48.231726 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.231696 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:24:48.248651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.248617 4431 generic.go:356] "Generic (PLEG): container finished" podID="0af70ae2-6fdb-455d-89f8-024f6003f52e" containerID="c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9" exitCode=0 Oct 30 06:24:48.248789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.248729 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" Oct 30 06:24:48.248789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.248763 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" event={"ID":"0af70ae2-6fdb-455d-89f8-024f6003f52e","Type":"ContainerDied","Data":"c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9"} Oct 30 06:24:48.248891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.248818 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f" event={"ID":"0af70ae2-6fdb-455d-89f8-024f6003f52e","Type":"ContainerDied","Data":"60a3fcf34593d720eb0be6d947288e6052ae70b4dfd1dc7059a4dd793602fd1c"} Oct 30 06:24:48.248891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.248841 4431 scope.go:122] "RemoveContainer" containerID="c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9" Oct 30 06:24:48.258958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.258935 4431 scope.go:122] "RemoveContainer" containerID="c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9" Oct 30 06:24:48.259269 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:24:48.259250 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9\": container with ID starting with c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9 not found: ID does not exist" containerID="c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9" Oct 30 06:24:48.259352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.259279 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9"} err="failed to get container status \"c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9\": rpc error: code = NotFound desc = could not find container \"c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9\": container with ID starting with c46bee4e641ae28aac925e0a2c9c9db16a18c37fbe43ee21b272b10fbd039ea9 not found: ID does not exist" Oct 30 06:24:48.294893 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.294856 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-maas-api-tls\") pod \"0af70ae2-6fdb-455d-89f8-024f6003f52e\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " Oct 30 06:24:48.294893 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.294896 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af70ae2-6fdb-455d-89f8-024f6003f52e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af70ae2-6fdb-455d-89f8-024f6003f52e-openshift-service-ca\") pod \"0af70ae2-6fdb-455d-89f8-024f6003f52e\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " Oct 30 06:24:48.295177 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.294992 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-metrics-certs\") pod \"0af70ae2-6fdb-455d-89f8-024f6003f52e\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " Oct 30 06:24:48.295250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.295185 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af70ae2-6fdb-455d-89f8-024f6003f52e-kube-api-access-zrg4x\" (UniqueName: \"kubernetes.io/projected/0af70ae2-6fdb-455d-89f8-024f6003f52e-kube-api-access-zrg4x\") pod \"0af70ae2-6fdb-455d-89f8-024f6003f52e\" (UID: \"0af70ae2-6fdb-455d-89f8-024f6003f52e\") " Oct 30 06:24:48.295436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.295403 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af70ae2-6fdb-455d-89f8-024f6003f52e-openshift-service-ca" pod "0af70ae2-6fdb-455d-89f8-024f6003f52e" (UID: "0af70ae2-6fdb-455d-89f8-024f6003f52e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:48.295738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.295716 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af70ae2-6fdb-455d-89f8-024f6003f52e-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:48.297574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.297549 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-metrics-certs" pod "0af70ae2-6fdb-455d-89f8-024f6003f52e" (UID: "0af70ae2-6fdb-455d-89f8-024f6003f52e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:48.297817 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.297797 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-maas-api-tls" pod "0af70ae2-6fdb-455d-89f8-024f6003f52e" (UID: "0af70ae2-6fdb-455d-89f8-024f6003f52e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:48.297957 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.297930 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af70ae2-6fdb-455d-89f8-024f6003f52e-kube-api-access-zrg4x" pod "0af70ae2-6fdb-455d-89f8-024f6003f52e" (UID: "0af70ae2-6fdb-455d-89f8-024f6003f52e"). InnerVolumeSpecName "kube-api-access-zrg4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:48.396638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.396590 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:48.396638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.396630 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af70ae2-6fdb-455d-89f8-024f6003f52e-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:48.396638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.396646 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrg4x\" (UniqueName: \"kubernetes.io/projected/0af70ae2-6fdb-455d-89f8-024f6003f52e-kube-api-access-zrg4x\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:24:48.605022 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.604975 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f"] Oct 30 06:24:48.626029 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:48.625990 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f2cf8a-f447d9848-2gr7f"] Oct 30 06:24:49.357539 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:49.357500 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af70ae2-6fdb-455d-89f8-024f6003f52e" path="/var/lib/kubelet/pods/0af70ae2-6fdb-455d-89f8-024f6003f52e/volumes" Oct 30 06:24:52.425235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.425194 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2"] Oct 30 06:24:52.425812 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.425795 4431 cpu_manager.go:414] "containerMap: removing container" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" Oct 30 06:24:52.425858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.425814 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84beb3ad-bff6-4552-b721-dc7a05b4cc92" containerName="istio-proxy" Oct 30 06:24:52.425858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.425850 4431 cpu_manager.go:414] "containerMap: removing container" podUID="0af70ae2-6fdb-455d-89f8-024f6003f52e" containerName="maas-api" Oct 30 06:24:52.425858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.425856 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af70ae2-6fdb-455d-89f8-024f6003f52e" containerName="maas-api" Oct 30 06:24:52.425961 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.425952 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af70ae2-6fdb-455d-89f8-024f6003f52e" containerName="maas-api" Oct 30 06:24:52.430868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.430843 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.452269 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.452237 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2"] Oct 30 06:24:52.568883 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.568822 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.569095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.568895 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.569095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.569031 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.569183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.569140 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.569183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.569176 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.569283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.569268 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.569374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.569360 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpb6w\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-kube-api-access-qpb6w\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.569423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.569412 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.569460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.569442 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.670691 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.670647 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.670927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.670705 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.670927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.670758 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.670927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.670802 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.670927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.670865 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.671146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.670940 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.671146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.670977 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.671146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.671022 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.671146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.671109 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpb6w\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-kube-api-access-qpb6w\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.671888 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.671853 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.672075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.671941 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.672075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.671996 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.672075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.672044 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.672574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.672542 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.674240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.674213 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.674702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.674678 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.699113 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.699075 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.699278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.699181 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpb6w\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-kube-api-access-qpb6w\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:52.746739 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:52.746699 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:53.065344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:53.065294 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2"] Oct 30 06:24:53.065907 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:24:53.065878 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcee06a3_8322_4bc6_86bf_d2e346f1fa9b.slice/crio-2d3c8a5416035601acbc77cde44fd23ba6df954ae722838f47033c5a29325afb WatchSource:0}: Error finding container 2d3c8a5416035601acbc77cde44fd23ba6df954ae722838f47033c5a29325afb: Status 404 returned error can't find the container with id 2d3c8a5416035601acbc77cde44fd23ba6df954ae722838f47033c5a29325afb Oct 30 06:24:53.069352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:53.069282 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:24:53.069438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:53.069418 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:24:53.069491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:53.069475 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:24:53.283231 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:53.283187 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" event={"ID":"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b","Type":"ContainerStarted","Data":"323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca"} Oct 30 06:24:53.283453 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:53.283252 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" event={"ID":"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b","Type":"ContainerStarted","Data":"2d3c8a5416035601acbc77cde44fd23ba6df954ae722838f47033c5a29325afb"} Oct 30 06:24:53.329038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:53.328920 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" podStartSLOduration=1.328900082 podStartE2EDuration="1.328900082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:53.327245884 +0000 UTC m=+1902.133767632" watchObservedRunningTime="2026-10-30 06:24:53.328900082 +0000 UTC m=+1902.135421904" Oct 30 06:24:53.747642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:53.747598 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:24:54.749294 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:54.749250 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 06:24:54.749766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:54.749346 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 06:24:55.747614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:55.747571 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 06:24:55.747949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:55.747642 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 06:24:56.747427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:56.747379 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 06:24:56.747984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:56.747448 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 06:24:57.747899 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:57.747867 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 06:24:57.748337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:57.747918 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 06:24:58.747348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:58.747289 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 06:24:58.747542 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:58.747369 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 06:24:59.747741 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:59.747700 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 06:24:59.748239 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:24:59.747767 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 06:25:00.856840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:00.856802 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:25:00.857289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:00.857239 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:25:00.858289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:00.858265 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:25:01.836976 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:01.836934 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq"] Oct 30 06:25:01.837219 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:01.837191 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" containerID="cri-o://400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a" gracePeriod=30 Oct 30 06:25:03.701684 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:25:03.701647 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:25:04.646374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:04.646334 4431 patch_prober.go:29] interesting pod/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:25:04.646582 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:04.646405 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:25:07.107579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.107503 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:25:07.177067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177038 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-token\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-token\") pod \"a7ab613a-ebe8-40c9-88f6-559d74969539\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " Oct 30 06:25:07.177247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177080 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ab613a-ebe8-40c9-88f6-559d74969539-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab613a-ebe8-40c9-88f6-559d74969539-istiod-ca-cert\") pod \"a7ab613a-ebe8-40c9-88f6-559d74969539\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " Oct 30 06:25:07.177247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177116 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-socket\") pod \"a7ab613a-ebe8-40c9-88f6-559d74969539\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " Oct 30 06:25:07.177247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177181 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-data\") pod \"a7ab613a-ebe8-40c9-88f6-559d74969539\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " Oct 30 06:25:07.177247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177226 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-credential-socket\") pod \"a7ab613a-ebe8-40c9-88f6-559d74969539\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " Oct 30 06:25:07.177508 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177260 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-kube-api-access-dghf9\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-kube-api-access-dghf9\") pod \"a7ab613a-ebe8-40c9-88f6-559d74969539\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " Oct 30 06:25:07.177508 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177293 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-podinfo\") pod \"a7ab613a-ebe8-40c9-88f6-559d74969539\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " Oct 30 06:25:07.177508 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177395 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-certs\") pod \"a7ab613a-ebe8-40c9-88f6-559d74969539\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " Oct 30 06:25:07.177752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177508 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-envoy\") pod \"a7ab613a-ebe8-40c9-88f6-559d74969539\" (UID: \"a7ab613a-ebe8-40c9-88f6-559d74969539\") " Oct 30 06:25:07.177752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177603 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-certs" pod "a7ab613a-ebe8-40c9-88f6-559d74969539" (UID: "a7ab613a-ebe8-40c9-88f6-559d74969539"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:07.177752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177635 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-credential-socket" pod "a7ab613a-ebe8-40c9-88f6-559d74969539" (UID: "a7ab613a-ebe8-40c9-88f6-559d74969539"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:07.177903 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.177846 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-data" pod "a7ab613a-ebe8-40c9-88f6-559d74969539" (UID: "a7ab613a-ebe8-40c9-88f6-559d74969539"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:07.178233 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.178172 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-socket" pod "a7ab613a-ebe8-40c9-88f6-559d74969539" (UID: "a7ab613a-ebe8-40c9-88f6-559d74969539"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:07.178233 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.178197 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:25:07.178233 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.178218 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:25:07.178233 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.178234 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:25:07.178570 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.178382 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ab613a-ebe8-40c9-88f6-559d74969539-istiod-ca-cert" pod "a7ab613a-ebe8-40c9-88f6-559d74969539" (UID: "a7ab613a-ebe8-40c9-88f6-559d74969539"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:07.179917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.179887 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-token" pod "a7ab613a-ebe8-40c9-88f6-559d74969539" (UID: "a7ab613a-ebe8-40c9-88f6-559d74969539"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:07.180035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.179993 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-kube-api-access-dghf9" pod "a7ab613a-ebe8-40c9-88f6-559d74969539" (UID: "a7ab613a-ebe8-40c9-88f6-559d74969539"). InnerVolumeSpecName "kube-api-access-dghf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:07.180262 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.180235 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-podinfo" pod "a7ab613a-ebe8-40c9-88f6-559d74969539" (UID: "a7ab613a-ebe8-40c9-88f6-559d74969539"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:25:07.180763 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.180740 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-envoy" pod "a7ab613a-ebe8-40c9-88f6-559d74969539" (UID: "a7ab613a-ebe8-40c9-88f6-559d74969539"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:07.279991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.279952 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dghf9\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-kube-api-access-dghf9\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:25:07.279991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.279985 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:25:07.279991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.279996 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:25:07.280219 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.280004 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ab613a-ebe8-40c9-88f6-559d74969539-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:25:07.280219 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.280012 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab613a-ebe8-40c9-88f6-559d74969539-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:25:07.280219 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.280020 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ab613a-ebe8-40c9-88f6-559d74969539-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:25:07.377980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.377899 4431 generic.go:356] "Generic (PLEG): container finished" podID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerID="400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a" exitCode=0 Oct 30 06:25:07.377980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.377952 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" Oct 30 06:25:07.378159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.377983 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" event={"ID":"a7ab613a-ebe8-40c9-88f6-559d74969539","Type":"ContainerDied","Data":"400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a"} Oct 30 06:25:07.378159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.378044 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq" event={"ID":"a7ab613a-ebe8-40c9-88f6-559d74969539","Type":"ContainerDied","Data":"f4c5b7114c1f8c87b15f57ad3308f2a43f0134e4158b9e3e6c6cdf1e8e4312ad"} Oct 30 06:25:07.378159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.378075 4431 scope.go:122] "RemoveContainer" containerID="400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a" Oct 30 06:25:07.387695 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.387672 4431 scope.go:122] "RemoveContainer" containerID="2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17" Oct 30 06:25:07.397457 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.397059 4431 scope.go:122] "RemoveContainer" containerID="400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a" Oct 30 06:25:07.397457 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:25:07.397390 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a\": container with ID starting with 400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a not found: ID does not exist" containerID="400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a" Oct 30 06:25:07.397457 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.397419 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a"} err="failed to get container status \"400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a\": rpc error: code = NotFound desc = could not find container \"400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a\": container with ID starting with 400470d1d4b1a3d27996b23b998801a8bb2a1197cf092206d41853673d2e3b3a not found: ID does not exist" Oct 30 06:25:07.397457 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.397441 4431 scope.go:122] "RemoveContainer" containerID="2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17" Oct 30 06:25:07.398443 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:25:07.397695 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17\": container with ID starting with 2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17 not found: ID does not exist" containerID="2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17" Oct 30 06:25:07.398443 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.397721 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17"} err="failed to get container status \"2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17\": rpc error: code = NotFound desc = could not find container \"2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17\": container with ID starting with 2a14cd19cc8c8c270fa7bf3ee741bc14a8f78e2679e7fce0e664012b89f3ca17 not found: ID does not exist" Oct 30 06:25:07.432005 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.431978 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq"] Oct 30 06:25:07.437322 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:07.437287 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a035a3-openshift-default-799cf89fc-clmxq"] Oct 30 06:25:09.338509 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:09.338468 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" path="/var/lib/kubelet/pods/a7ab613a-ebe8-40c9-88f6-559d74969539/volumes" Oct 30 06:25:33.703895 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:25:33.703856 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:25:40.054999 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.054943 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj"] Oct 30 06:25:40.055559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.055544 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" Oct 30 06:25:40.055616 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.055562 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" Oct 30 06:25:40.055731 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.055716 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" Oct 30 06:25:40.059097 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.059076 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.082912 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.082873 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj"] Oct 30 06:25:40.138910 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.138871 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.139138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.138931 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.139138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.139078 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.139278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.139257 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.139367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.139346 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82642864-9423-44f0-b32d-4de4b1cf3e2c-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.139497 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.139433 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.139497 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.139466 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.139572 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.139531 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.139628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.139592 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq8r6\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-kube-api-access-rq8r6\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.240555 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.240515 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq8r6\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-kube-api-access-rq8r6\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.240741 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.240607 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.240741 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.240656 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.240741 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.240692 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.240891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.240816 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.240891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.240843 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82642864-9423-44f0-b32d-4de4b1cf3e2c-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.240990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.240895 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.240990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.240926 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.240990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.240952 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.241260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.241235 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.241489 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.241416 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.241604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.241581 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.241661 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.241603 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.241944 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.241919 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82642864-9423-44f0-b32d-4de4b1cf3e2c-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.243141 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.243114 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.243494 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.243475 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.267942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.267908 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.268099 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.267964 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq8r6\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-kube-api-access-rq8r6\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.375617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.375515 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:40.704814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.704776 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj"] Oct 30 06:25:40.716486 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:25:40.716457 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82642864_9423_44f0_b32d_4de4b1cf3e2c.slice/crio-13420930935e677a06dfdff79658fabd8125c32cb173400a39ec2d9f7a388fc8 WatchSource:0}: Error finding container 13420930935e677a06dfdff79658fabd8125c32cb173400a39ec2d9f7a388fc8: Status 404 returned error can't find the container with id 13420930935e677a06dfdff79658fabd8125c32cb173400a39ec2d9f7a388fc8 Oct 30 06:25:40.719005 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.718968 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:25:40.719123 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.719044 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:25:40.719123 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.719075 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:25:40.797258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.797217 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" event={"ID":"82642864-9423-44f0-b32d-4de4b1cf3e2c","Type":"ContainerStarted","Data":"fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44"} Oct 30 06:25:40.797461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.797288 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" event={"ID":"82642864-9423-44f0-b32d-4de4b1cf3e2c","Type":"ContainerStarted","Data":"13420930935e677a06dfdff79658fabd8125c32cb173400a39ec2d9f7a388fc8"} Oct 30 06:25:40.845268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:40.845195 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" podStartSLOduration=0.845172758 podStartE2EDuration="845.172758ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:40.842094587 +0000 UTC m=+1949.648616340" watchObservedRunningTime="2026-10-30 06:25:40.845172758 +0000 UTC m=+1949.651694512" Oct 30 06:25:41.384330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:41.378379 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:41.384330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:41.378565 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:25:41.384330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:41.378613 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 06:25:42.385902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:42.380761 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:25:42.385902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:42.380849 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 06:25:43.384328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:43.383088 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:25:43.384328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:43.383151 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 06:25:44.377464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:44.376811 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:25:44.377464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:44.376934 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 06:25:45.384801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:45.384764 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:25:45.385344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:45.384828 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 06:25:46.383816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:46.383770 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:25:46.384114 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:46.384084 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 06:25:47.376250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:47.376217 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 06:25:47.376802 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:47.376276 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 06:25:48.380980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:48.380946 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:48.381434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:48.381206 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:25:48.382046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:25:48.382028 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:26:03.701863 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:26:03.701820 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:26:07.233882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.233841 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s"] Oct 30 06:26:07.234640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.234622 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" Oct 30 06:26:07.234701 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.234640 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" Oct 30 06:26:07.234772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.234762 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ab613a-ebe8-40c9-88f6-559d74969539" containerName="istio-proxy" Oct 30 06:26:07.238035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.238018 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.266474 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.266433 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s"] Oct 30 06:26:07.326534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.326496 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.326719 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.326559 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.326719 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.326682 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.326719 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.326710 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.326912 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.326890 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.326953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.326921 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d697083f-5df2-477f-8342-74aafc9e2343-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.326994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.326948 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879xm\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-kube-api-access-879xm\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.327067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.327056 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.327109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.327083 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d697083f-5df2-477f-8342-74aafc9e2343-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.428751 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.428709 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.428751 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.428748 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.429035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.428796 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.429035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.428813 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d697083f-5df2-477f-8342-74aafc9e2343-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.429035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.428850 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879xm\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-kube-api-access-879xm\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.429035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.428935 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.429035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.428973 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d697083f-5df2-477f-8342-74aafc9e2343-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.429357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.429044 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.429357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.429092 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.429586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.429554 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.429767 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.429649 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.430070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.430043 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.430557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.430096 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d697083f-5df2-477f-8342-74aafc9e2343-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.430557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.430389 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.435045 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.435005 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d697083f-5df2-477f-8342-74aafc9e2343-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.435834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.435809 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.452694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.452654 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.453984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.453962 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-879xm\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-kube-api-access-879xm\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.552785 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.552697 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:07.874978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.874943 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s"] Oct 30 06:26:07.875393 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:26:07.875357 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd697083f_5df2_477f_8342_74aafc9e2343.slice/crio-a38a0d3dcdb5d0c0b3aa2b9b56b6c385a3b2eda9c8b89f4790da180323596fc0 WatchSource:0}: Error finding container a38a0d3dcdb5d0c0b3aa2b9b56b6c385a3b2eda9c8b89f4790da180323596fc0: Status 404 returned error can't find the container with id a38a0d3dcdb5d0c0b3aa2b9b56b6c385a3b2eda9c8b89f4790da180323596fc0 Oct 30 06:26:07.878090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.878054 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:26:07.878188 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.878130 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:26:07.878188 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:07.878158 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:26:08.139532 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:08.139429 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" event={"ID":"d697083f-5df2-477f-8342-74aafc9e2343","Type":"ContainerStarted","Data":"2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6"} Oct 30 06:26:08.139532 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:08.139493 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" event={"ID":"d697083f-5df2-477f-8342-74aafc9e2343","Type":"ContainerStarted","Data":"a38a0d3dcdb5d0c0b3aa2b9b56b6c385a3b2eda9c8b89f4790da180323596fc0"} Oct 30 06:26:08.191134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:08.191075 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" podStartSLOduration=1.191059257 podStartE2EDuration="1.191059257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:08.190380271 +0000 UTC m=+1976.996902021" watchObservedRunningTime="2026-10-30 06:26:08.191059257 +0000 UTC m=+1976.997581005" Oct 30 06:26:08.552967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:08.552924 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:08.554937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:08.554908 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 06:26:08.555082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:08.554989 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 06:26:09.553484 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:09.553452 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 06:26:09.554006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:09.553504 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 06:26:10.553574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:10.553527 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 06:26:10.553997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:10.553589 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 06:26:11.554174 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:11.554128 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 06:26:11.554653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:11.554191 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 06:26:12.553390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:12.553351 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 06:26:12.553592 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:12.553415 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 06:26:13.553450 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:13.553398 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 06:26:13.554055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:13.553472 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 06:26:14.553184 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:14.553136 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 06:26:14.553407 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:14.553216 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 06:26:15.661860 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:15.661279 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:15.664581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:15.661944 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:15.664581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:15.663042 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:26:22.293145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.293109 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp"] Oct 30 06:26:22.298347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.298322 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.323681 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.323645 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp"] Oct 30 06:26:22.341875 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.341829 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.342103 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.341902 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.342103 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.342037 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnjw\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-kube-api-access-2vnjw\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.342103 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.342087 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.342296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.342170 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.342296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.342199 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.342296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.342275 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.342487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.342326 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.342487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.342374 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.443551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443515 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.443749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443605 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnjw\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-kube-api-access-2vnjw\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.443749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443639 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.443749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443678 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.443749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443710 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.443970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443755 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.443970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443791 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.443970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443833 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.443970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443929 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.444173 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.443980 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.444460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.444422 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.444611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.444577 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.444611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.444600 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.444832 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.444812 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.446214 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.446196 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.446626 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.446604 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.467383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.467342 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.467383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.467383 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnjw\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-kube-api-access-2vnjw\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.614859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.614759 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:22.941954 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.941919 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp"] Oct 30 06:26:22.945538 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.945149 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:26:22.945538 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.945280 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:26:22.945688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:22.945401 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:26:23.253297 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:23.253256 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" event={"ID":"b6bdf7fe-250e-4998-90ad-cea9238b3ce9","Type":"ContainerStarted","Data":"535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757"} Oct 30 06:26:23.253480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:23.253327 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" event={"ID":"b6bdf7fe-250e-4998-90ad-cea9238b3ce9","Type":"ContainerStarted","Data":"9c02bc716d27f2a944da8f7479446259a30742aaee777ee406b3bb345b6a1444"} Oct 30 06:26:23.303689 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:23.303626 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" podStartSLOduration=1.30360569 podStartE2EDuration="1.30360569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:23.299042767 +0000 UTC m=+1992.105564516" watchObservedRunningTime="2026-10-30 06:26:23.30360569 +0000 UTC m=+1992.110127439" Oct 30 06:26:23.615362 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:23.615250 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:24.617038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:24.617001 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 06:26:24.617478 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:24.617057 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 06:26:25.616245 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:25.616195 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 06:26:25.616450 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:25.616275 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 06:26:26.615182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:26.615143 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 06:26:26.615742 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:26.615270 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 06:26:27.615908 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:27.615869 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 06:26:27.616368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:27.615928 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 06:26:28.615693 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:28.615650 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 06:26:28.615880 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:28.615716 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 06:26:29.615614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:29.615564 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 06:26:29.616270 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:29.616232 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 06:26:30.695745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:30.695704 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:30.696245 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:30.696178 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:30.697129 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:30.697107 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:26:33.702753 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:26:33.702716 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:26:53.127218 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.127181 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc"] Oct 30 06:26:53.132327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.132289 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.157546 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.157497 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-4f3973-openshift-default-dockercfg-4xk9d\"" Oct 30 06:26:53.160773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.160666 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc"] Oct 30 06:26:53.244592 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.244557 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6h5m\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-kube-api-access-d6h5m\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.244783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.244602 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-envoy\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.244783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.244625 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d865d5a-0560-4d6e-ae24-6985104463f8-istiod-ca-cert\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.244783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.244691 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-token\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.244783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.244760 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-data\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.244963 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.244896 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-credential-socket\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.244963 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.244925 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-socket\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.244963 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.244958 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-podinfo\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.245070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.245013 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-certs\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.349394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.349348 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6h5m\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-kube-api-access-d6h5m\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.349597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.349417 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-envoy\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.349597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.349447 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d865d5a-0560-4d6e-ae24-6985104463f8-istiod-ca-cert\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.349597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.349475 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-token\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.349597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.349517 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-data\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.349775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.349619 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-credential-socket\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.349775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.349649 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-socket\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.349775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.349691 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-podinfo\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.349775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.349751 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-certs\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.350329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.350273 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-certs\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.352618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.352435 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-credential-socket\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.353347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.353321 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-socket\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.353513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.353385 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d865d5a-0560-4d6e-ae24-6985104463f8-istiod-ca-cert\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.353666 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.353641 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-data\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.361644 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.361583 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-envoy\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.362193 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.362141 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-podinfo\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.402367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.402197 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6h5m\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-kube-api-access-d6h5m\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.409408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.409295 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-token\") pod \"e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.449554 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.449521 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:53.784393 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:26:53.784360 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d865d5a_0560_4d6e_ae24_6985104463f8.slice/crio-eece24c410da84645e9210c31bb20df44aa7ff52769dfa7e822f3c338e198e1e WatchSource:0}: Error finding container eece24c410da84645e9210c31bb20df44aa7ff52769dfa7e822f3c338e198e1e: Status 404 returned error can't find the container with id eece24c410da84645e9210c31bb20df44aa7ff52769dfa7e822f3c338e198e1e Oct 30 06:26:53.784727 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.784705 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc"] Oct 30 06:26:53.786811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.786775 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:26:53.786945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.786875 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:26:53.786945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:53.786937 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:26:54.532878 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:54.532840 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" event={"ID":"6d865d5a-0560-4d6e-ae24-6985104463f8","Type":"ContainerStarted","Data":"8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be"} Oct 30 06:26:54.533526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:54.533498 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" event={"ID":"6d865d5a-0560-4d6e-ae24-6985104463f8","Type":"ContainerStarted","Data":"eece24c410da84645e9210c31bb20df44aa7ff52769dfa7e822f3c338e198e1e"} Oct 30 06:26:54.597803 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:54.597707 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" podStartSLOduration=1.597667465 podStartE2EDuration="1.597667465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:54.591076666 +0000 UTC m=+2023.397598414" watchObservedRunningTime="2026-10-30 06:26:54.597667465 +0000 UTC m=+2023.404189214" Oct 30 06:26:55.449975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:55.449932 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:55.455597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:55.455564 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:55.540368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:55.540226 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:26:55.541086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:26:55.541066 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:27:03.702570 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:27:03.702527 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:27:24.101364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.101325 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz"] Oct 30 06:27:24.110676 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.110646 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:24.134324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.130746 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-30fbb6-dockercfg-75j5s\"" Oct 30 06:27:24.134324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.130838 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-30fbb6\"" Oct 30 06:27:24.152655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.149521 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz"] Oct 30 06:27:24.171222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.166108 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62504278-ebeb-48fd-a9a3-6a68dff619e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz\" (UID: \"62504278-ebeb-48fd-a9a3-6a68dff619e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:24.171222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.166177 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2r5\" (UniqueName: \"kubernetes.io/projected/62504278-ebeb-48fd-a9a3-6a68dff619e4-kube-api-access-cb2r5\") pod \"payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz\" (UID: \"62504278-ebeb-48fd-a9a3-6a68dff619e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:24.267080 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.267039 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62504278-ebeb-48fd-a9a3-6a68dff619e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz\" (UID: \"62504278-ebeb-48fd-a9a3-6a68dff619e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:24.267280 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.267103 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2r5\" (UniqueName: \"kubernetes.io/projected/62504278-ebeb-48fd-a9a3-6a68dff619e4-kube-api-access-cb2r5\") pod \"payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz\" (UID: \"62504278-ebeb-48fd-a9a3-6a68dff619e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:24.267780 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.267759 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62504278-ebeb-48fd-a9a3-6a68dff619e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz\" (UID: \"62504278-ebeb-48fd-a9a3-6a68dff619e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:24.290807 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.290769 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2r5\" (UniqueName: \"kubernetes.io/projected/62504278-ebeb-48fd-a9a3-6a68dff619e4-kube-api-access-cb2r5\") pod \"payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz\" (UID: \"62504278-ebeb-48fd-a9a3-6a68dff619e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:24.451875 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.451837 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:24.757764 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:24.757730 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz"] Oct 30 06:27:24.758926 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:27:24.758899 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62504278_ebeb_48fd_a9a3_6a68dff619e4.slice/crio-90282d61542e2774f90aea17b84b7e196c421caaff20eaa5777b08a59944e614 WatchSource:0}: Error finding container 90282d61542e2774f90aea17b84b7e196c421caaff20eaa5777b08a59944e614: Status 404 returned error can't find the container with id 90282d61542e2774f90aea17b84b7e196c421caaff20eaa5777b08a59944e614 Oct 30 06:27:25.746865 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:25.746821 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" event={"ID":"62504278-ebeb-48fd-a9a3-6a68dff619e4","Type":"ContainerStarted","Data":"80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142"} Oct 30 06:27:25.747383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:25.746890 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" event={"ID":"62504278-ebeb-48fd-a9a3-6a68dff619e4","Type":"ContainerStarted","Data":"90282d61542e2774f90aea17b84b7e196c421caaff20eaa5777b08a59944e614"} Oct 30 06:27:25.747383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:25.747019 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:25.788408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:25.788266 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" podStartSLOduration=1.788250602 podStartE2EDuration="1.788250602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:25.78775563 +0000 UTC m=+2054.594277379" watchObservedRunningTime="2026-10-30 06:27:25.788250602 +0000 UTC m=+2054.594772349" Oct 30 06:27:33.722421 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:27:33.722342 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:27:36.756570 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:36.756540 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 06:27:37.954910 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:37.954874 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7"] Oct 30 06:27:37.959594 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:37.959563 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:37.977423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:37.977394 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7"] Oct 30 06:27:37.985067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:37.985039 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-868b1f\"" Oct 30 06:27:38.057884 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.057853 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pr7g\" (UniqueName: \"kubernetes.io/projected/4b4385bc-6024-4651-8422-f785697b8049-kube-api-access-8pr7g\") pod \"payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7\" (UID: \"4b4385bc-6024-4651-8422-f785697b8049\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:38.058069 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.057932 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4385bc-6024-4651-8422-f785697b8049-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7\" (UID: \"4b4385bc-6024-4651-8422-f785697b8049\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:38.159123 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.159079 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pr7g\" (UniqueName: \"kubernetes.io/projected/4b4385bc-6024-4651-8422-f785697b8049-kube-api-access-8pr7g\") pod \"payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7\" (UID: \"4b4385bc-6024-4651-8422-f785697b8049\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:38.159357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.159190 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4385bc-6024-4651-8422-f785697b8049-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7\" (UID: \"4b4385bc-6024-4651-8422-f785697b8049\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:38.159970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.159949 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4385bc-6024-4651-8422-f785697b8049-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7\" (UID: \"4b4385bc-6024-4651-8422-f785697b8049\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:38.182748 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.182716 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pr7g\" (UniqueName: \"kubernetes.io/projected/4b4385bc-6024-4651-8422-f785697b8049-kube-api-access-8pr7g\") pod \"payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7\" (UID: \"4b4385bc-6024-4651-8422-f785697b8049\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:38.277996 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.277593 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:38.637014 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:27:38.636982 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4385bc_6024_4651_8422_f785697b8049.slice/crio-ab227889f163f6fd6103a2dc7de21cf3b86851c82a35c7438626eccb0fca91b4 WatchSource:0}: Error finding container ab227889f163f6fd6103a2dc7de21cf3b86851c82a35c7438626eccb0fca91b4: Status 404 returned error can't find the container with id ab227889f163f6fd6103a2dc7de21cf3b86851c82a35c7438626eccb0fca91b4 Oct 30 06:27:38.637143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.637078 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7"] Oct 30 06:27:38.835229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.835193 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" event={"ID":"4b4385bc-6024-4651-8422-f785697b8049","Type":"ContainerStarted","Data":"ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89"} Oct 30 06:27:38.835450 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.835251 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" event={"ID":"4b4385bc-6024-4651-8422-f785697b8049","Type":"ContainerStarted","Data":"ab227889f163f6fd6103a2dc7de21cf3b86851c82a35c7438626eccb0fca91b4"} Oct 30 06:27:38.835450 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.835281 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:38.875168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:38.875106 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" podStartSLOduration=1.8750913630000001 podStartE2EDuration="1.875091363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:38.874903427 +0000 UTC m=+2067.681425175" watchObservedRunningTime="2026-10-30 06:27:38.875091363 +0000 UTC m=+2067.681613105" Oct 30 06:27:49.047356 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.044705 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9"] Oct 30 06:27:49.050315 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.050268 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.072349 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.072314 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9"] Oct 30 06:27:49.085657 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.085623 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-b9ff93-openshift-default-dockercfg-5zp4s\"" Oct 30 06:27:49.167430 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.167385 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmhjm\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-kube-api-access-vmhjm\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.167633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.167450 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-envoy\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.167633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.167489 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-data\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.167773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.167649 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-certs\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.167773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.167719 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-podinfo\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.167773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.167757 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54aa0ffd-3489-4d17-9171-f56a8664045c-istiod-ca-cert\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.167975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.167956 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-socket\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.168047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.168003 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-token\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.168150 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.168124 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-credential-socket\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.271324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.271139 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmhjm\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-kube-api-access-vmhjm\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.271324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.271196 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-envoy\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.271324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.271220 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-data\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.271324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.271263 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-certs\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.271324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.271321 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-podinfo\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.271753 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.271354 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54aa0ffd-3489-4d17-9171-f56a8664045c-istiod-ca-cert\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.271753 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.271527 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-socket\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.271753 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.271576 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-token\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.271753 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.271626 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-credential-socket\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.272368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.272082 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-credential-socket\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.275727 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.275221 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54aa0ffd-3489-4d17-9171-f56a8664045c-istiod-ca-cert\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.277040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.276207 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-socket\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.277040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.276512 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-data\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.280372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.280344 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-certs\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.293712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.293637 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-envoy\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.309188 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.309097 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-token\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.316417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.313083 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmhjm\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-kube-api-access-vmhjm\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.323417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.323344 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-podinfo\") pod \"e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.371675 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.371636 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:49.714810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.714774 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9"] Oct 30 06:27:49.715362 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:27:49.715337 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54aa0ffd_3489_4d17_9171_f56a8664045c.slice/crio-fdc6b0a98acf6fbb4e4b080be76206a2cd717b3b5cc4ebe2f51de8726bde1f00 WatchSource:0}: Error finding container fdc6b0a98acf6fbb4e4b080be76206a2cd717b3b5cc4ebe2f51de8726bde1f00: Status 404 returned error can't find the container with id fdc6b0a98acf6fbb4e4b080be76206a2cd717b3b5cc4ebe2f51de8726bde1f00 Oct 30 06:27:49.718600 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.718566 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:27:49.718672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.718638 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:27:49.718672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.718667 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:27:49.844810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.844763 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 06:27:49.935207 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.935162 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" event={"ID":"54aa0ffd-3489-4d17-9171-f56a8664045c","Type":"ContainerStarted","Data":"1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e"} Oct 30 06:27:49.935445 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.935240 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" event={"ID":"54aa0ffd-3489-4d17-9171-f56a8664045c","Type":"ContainerStarted","Data":"fdc6b0a98acf6fbb4e4b080be76206a2cd717b3b5cc4ebe2f51de8726bde1f00"} Oct 30 06:27:49.980270 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:49.980152 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" podStartSLOduration=0.980133058 podStartE2EDuration="980.133058ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:49.979380993 +0000 UTC m=+2078.785902745" watchObservedRunningTime="2026-10-30 06:27:49.980133058 +0000 UTC m=+2078.786654808" Oct 30 06:27:50.371946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.371847 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:50.378391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.378361 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:50.943673 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.943572 4431 generic.go:356] "Generic (PLEG): container finished" podID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerID="8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be" exitCode=0 Oct 30 06:27:50.943673 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.943647 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" event={"ID":"6d865d5a-0560-4d6e-ae24-6985104463f8","Type":"ContainerDied","Data":"8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be"} Oct 30 06:27:50.944104 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.944084 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:50.944169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.944124 4431 scope.go:122] "RemoveContainer" containerID="8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be" Oct 30 06:27:50.945259 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.945168 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:27:50.946109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.946076 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:27:50.946183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.946168 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:27:50.946230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:50.946216 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:27:51.953355 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:51.953281 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" event={"ID":"6d865d5a-0560-4d6e-ae24-6985104463f8","Type":"ContainerStarted","Data":"8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530"} Oct 30 06:27:52.450001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:52.449960 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:27:52.451495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:52.451462 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 06:27:52.451653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:52.451541 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 06:27:53.450537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:53.450498 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 06:27:53.451046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:53.450576 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 06:27:54.449999 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:54.449963 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 06:27:54.450207 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:54.450024 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 06:27:55.450590 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:55.450543 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 06:27:55.451027 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:55.450625 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 06:27:55.540436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:55.540395 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:27:56.450374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:56.450333 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 06:27:56.450566 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:56.450399 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 06:27:57.454723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:57.454690 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:27:57.456014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:27:57.455988 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:28:03.702487 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:28:03.702450 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:28:33.707080 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:28:33.707043 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:28:44.185585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:44.185550 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8"] Oct 30 06:28:44.186028 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:44.185830 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" containerID="cri-o://a619164c0bc63cd10a713fd9ec2e0e36b911e728e993a32c5359e614bf97d0fc" gracePeriod=30 Oct 30 06:28:48.391139 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.391103 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx"] Oct 30 06:28:48.394825 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.394801 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.411611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.411581 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-fa5a69-openshift-default-dockercfg-6rbm2\"" Oct 30 06:28:48.432520 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.432480 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx"] Oct 30 06:28:48.507082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.507045 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.507082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.507088 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-data\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.507313 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.507180 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-envoy\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.507313 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.507234 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-certs\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.507313 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.507273 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvl9\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-kube-api-access-prvl9\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.507429 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.507389 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-podinfo\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.507527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.507512 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-token\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.507562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.507545 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20d2777-0ed1-4ead-a144-799c102a6d97-istiod-ca-cert\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.507617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.507583 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-credential-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.608448 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.608409 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-podinfo\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.608654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.608537 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-token\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.608654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.608577 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20d2777-0ed1-4ead-a144-799c102a6d97-istiod-ca-cert\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.608654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.608611 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-credential-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.608819 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.608767 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.608819 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.608801 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-data\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.608922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.608847 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-envoy\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.608922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.608883 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-certs\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.608922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.608910 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvl9\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-kube-api-access-prvl9\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.609100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.609064 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-credential-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.609171 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.609153 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-socket\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.609230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.609184 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-data\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.609344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.609322 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-certs\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.609482 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.609461 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20d2777-0ed1-4ead-a144-799c102a6d97-istiod-ca-cert\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.611026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.610999 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-envoy\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.611109 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.611054 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-podinfo\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.634875 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.634843 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-token\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.635081 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.635061 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvl9\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-kube-api-access-prvl9\") pod \"e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.709115 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.709037 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:48.830280 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.830243 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9jxv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:28:48.830458 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:48.830327 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:28:49.046466 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:28:49.046436 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20d2777_0ed1_4ead_a144_799c102a6d97.slice/crio-889ae609853b76515c4fc52a3f91350999b130315ebabaf4576c32f5d4bcc08f WatchSource:0}: Error finding container 889ae609853b76515c4fc52a3f91350999b130315ebabaf4576c32f5d4bcc08f: Status 404 returned error can't find the container with id 889ae609853b76515c4fc52a3f91350999b130315ebabaf4576c32f5d4bcc08f Oct 30 06:28:49.047286 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.047259 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx"] Oct 30 06:28:49.049051 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.049016 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:28:49.049152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.049082 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:28:49.049152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.049111 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:28:49.415531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.415491 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" event={"ID":"a20d2777-0ed1-4ead-a144-799c102a6d97","Type":"ContainerStarted","Data":"ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902"} Oct 30 06:28:49.416039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.415549 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" event={"ID":"a20d2777-0ed1-4ead-a144-799c102a6d97","Type":"ContainerStarted","Data":"889ae609853b76515c4fc52a3f91350999b130315ebabaf4576c32f5d4bcc08f"} Oct 30 06:28:49.418966 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.418642 4431 generic.go:356] "Generic (PLEG): container finished" podID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerID="a619164c0bc63cd10a713fd9ec2e0e36b911e728e993a32c5359e614bf97d0fc" exitCode=0 Oct 30 06:28:49.418966 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.418807 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" event={"ID":"b20f0e3c-d224-467b-8246-a93ade40a4da","Type":"ContainerDied","Data":"a619164c0bc63cd10a713fd9ec2e0e36b911e728e993a32c5359e614bf97d0fc"} Oct 30 06:28:49.473359 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.472232 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" podStartSLOduration=1.472212142 podStartE2EDuration="1.472212142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:49.470640897 +0000 UTC m=+2138.277162654" watchObservedRunningTime="2026-10-30 06:28:49.472212142 +0000 UTC m=+2138.278733892" Oct 30 06:28:49.519512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.519484 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.657007 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-data\") pod \"b20f0e3c-d224-467b-8246-a93ade40a4da\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.657096 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-credential-socket\") pod \"b20f0e3c-d224-467b-8246-a93ade40a4da\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.657173 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-podinfo\") pod \"b20f0e3c-d224-467b-8246-a93ade40a4da\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.657253 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-envoy\") pod \"b20f0e3c-d224-467b-8246-a93ade40a4da\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.657338 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-token\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-token\") pod \"b20f0e3c-d224-467b-8246-a93ade40a4da\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.657376 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20f0e3c-d224-467b-8246-a93ade40a4da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20f0e3c-d224-467b-8246-a93ade40a4da-istiod-ca-cert\") pod \"b20f0e3c-d224-467b-8246-a93ade40a4da\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.657441 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-kube-api-access-whv8l\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-kube-api-access-whv8l\") pod \"b20f0e3c-d224-467b-8246-a93ade40a4da\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.657560 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-certs\") pod \"b20f0e3c-d224-467b-8246-a93ade40a4da\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.657609 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-socket\") pod \"b20f0e3c-d224-467b-8246-a93ade40a4da\" (UID: \"b20f0e3c-d224-467b-8246-a93ade40a4da\") " Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.658766 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-socket" pod "b20f0e3c-d224-467b-8246-a93ade40a4da" (UID: "b20f0e3c-d224-467b-8246-a93ade40a4da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.660003 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-certs" pod "b20f0e3c-d224-467b-8246-a93ade40a4da" (UID: "b20f0e3c-d224-467b-8246-a93ade40a4da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.660349 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20f0e3c-d224-467b-8246-a93ade40a4da-istiod-ca-cert" pod "b20f0e3c-d224-467b-8246-a93ade40a4da" (UID: "b20f0e3c-d224-467b-8246-a93ade40a4da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.660413 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-credential-socket" pod "b20f0e3c-d224-467b-8246-a93ade40a4da" (UID: "b20f0e3c-d224-467b-8246-a93ade40a4da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:28:49.667384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.661697 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-data" pod "b20f0e3c-d224-467b-8246-a93ade40a4da" (UID: "b20f0e3c-d224-467b-8246-a93ade40a4da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:28:49.710351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.701552 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-kube-api-access-whv8l" pod "b20f0e3c-d224-467b-8246-a93ade40a4da" (UID: "b20f0e3c-d224-467b-8246-a93ade40a4da"). InnerVolumeSpecName "kube-api-access-whv8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:28:49.710351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.701659 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-envoy" pod "b20f0e3c-d224-467b-8246-a93ade40a4da" (UID: "b20f0e3c-d224-467b-8246-a93ade40a4da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:28:49.710351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.701718 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-podinfo" pod "b20f0e3c-d224-467b-8246-a93ade40a4da" (UID: "b20f0e3c-d224-467b-8246-a93ade40a4da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:28:49.711084 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.710668 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:49.712945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.712885 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-token" pod "b20f0e3c-d224-467b-8246-a93ade40a4da" (UID: "b20f0e3c-d224-467b-8246-a93ade40a4da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:28:49.759086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.758976 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:28:49.759086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.759003 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:28:49.759086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.759012 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20f0e3c-d224-467b-8246-a93ade40a4da-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:28:49.759086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.759022 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whv8l\" (UniqueName: \"kubernetes.io/projected/b20f0e3c-d224-467b-8246-a93ade40a4da-kube-api-access-whv8l\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:28:49.759086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.759031 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:28:49.759086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.759040 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:28:49.759086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.759048 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:28:49.759086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.759056 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20f0e3c-d224-467b-8246-a93ade40a4da-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:28:49.759086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:49.759063 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20f0e3c-d224-467b-8246-a93ade40a4da-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:28:50.430202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:50.430168 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" Oct 30 06:28:50.430202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:50.430180 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8" event={"ID":"b20f0e3c-d224-467b-8246-a93ade40a4da","Type":"ContainerDied","Data":"8dc5af640485004eb3ff0aed1fb152b43c8a5b6eac88e144494193dd2de11f82"} Oct 30 06:28:50.430778 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:50.430241 4431 scope.go:122] "RemoveContainer" containerID="a619164c0bc63cd10a713fd9ec2e0e36b911e728e993a32c5359e614bf97d0fc" Oct 30 06:28:50.519946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:50.519891 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8"] Oct 30 06:28:50.522717 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:50.522685 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9jxv8"] Oct 30 06:28:50.712933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:50.712885 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:28:50.712933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:50.712936 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 06:28:51.339440 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:51.339409 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" path="/var/lib/kubelet/pods/b20f0e3c-d224-467b-8246-a93ade40a4da/volumes" Oct 30 06:28:51.710767 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:51.710721 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:28:51.711388 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:51.710796 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 06:28:52.710209 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:52.710039 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:28:52.710209 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:52.710110 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 06:28:53.709864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:53.709825 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:28:53.710519 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:53.710472 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 06:28:54.709953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:54.709910 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:28:54.710506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:54.709971 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 06:28:55.710163 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:55.710125 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 06:28:55.710672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:55.710187 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 06:28:56.716028 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:56.715991 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:56.716656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:56.716428 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:56.717256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:56.717239 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 06:28:58.528677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.528631 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f"] Oct 30 06:28:58.529628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.529600 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" Oct 30 06:28:58.529628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.529629 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" Oct 30 06:28:58.529814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.529787 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20f0e3c-d224-467b-8246-a93ade40a4da" containerName="istio-proxy" Oct 30 06:28:58.535149 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.535123 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.550711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.550676 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f"] Oct 30 06:28:58.559534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.559503 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-30fbb6\"/\"auth-test-model-30fbb6-kserve-self-signed-certs\"" Oct 30 06:28:58.559689 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.559610 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-30fbb6\"/\"openshift-service-ca.crt\"" Oct 30 06:28:58.559732 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.559693 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-30fbb6\"/\"default-dockercfg-h5kn9\"" Oct 30 06:28:58.561915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.561891 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-30fbb6\"/\"kube-root-ca.crt\"" Oct 30 06:28:58.581006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.580960 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxtf8\" (UniqueName: \"kubernetes.io/projected/648f6993-0732-4d7f-96ec-dc9283e60c28-kube-api-access-bxtf8\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.581216 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.581110 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-tmp-dir\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.581216 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.581207 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-model-cache\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.581330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.581261 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648f6993-0732-4d7f-96ec-dc9283e60c28-tls-certs\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.581531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.581513 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-home\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.581588 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.581557 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-dshm\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.682758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.682718 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-model-cache\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.682758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.682758 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648f6993-0732-4d7f-96ec-dc9283e60c28-tls-certs\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.682988 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.682842 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-home\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.682988 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.682865 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-dshm\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.682988 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.682967 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxtf8\" (UniqueName: \"kubernetes.io/projected/648f6993-0732-4d7f-96ec-dc9283e60c28-kube-api-access-bxtf8\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.683124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.683047 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-tmp-dir\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.683345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.683297 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-model-cache\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.683587 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.683346 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-home\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.683587 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.683492 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-tmp-dir\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.685405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.685378 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-dshm\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.685496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.685480 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648f6993-0732-4d7f-96ec-dc9283e60c28-tls-certs\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.706178 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.706143 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxtf8\" (UniqueName: \"kubernetes.io/projected/648f6993-0732-4d7f-96ec-dc9283e60c28-kube-api-access-bxtf8\") pod \"auth-test-model-30fbb6-kserve-79665d489-hjh6f\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:58.849806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:58.849707 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:28:59.179542 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:28:59.179510 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod648f6993_0732_4d7f_96ec_dc9283e60c28.slice/crio-b43ea2d141c22bbfa37538cd10cedda16dd0f61ce10a5e53c124b375c1b8a8d5 WatchSource:0}: Error finding container b43ea2d141c22bbfa37538cd10cedda16dd0f61ce10a5e53c124b375c1b8a8d5: Status 404 returned error can't find the container with id b43ea2d141c22bbfa37538cd10cedda16dd0f61ce10a5e53c124b375c1b8a8d5 Oct 30 06:28:59.181001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:59.180970 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f"] Oct 30 06:28:59.490665 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:59.490633 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" event={"ID":"648f6993-0732-4d7f-96ec-dc9283e60c28","Type":"ContainerStarted","Data":"14f950d8b501f08a1d0404dbf9b0ca5069a3164ddbc49cc53a8e234cb9ff5792"} Oct 30 06:28:59.490856 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:59.490680 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" event={"ID":"648f6993-0732-4d7f-96ec-dc9283e60c28","Type":"ContainerStarted","Data":"b43ea2d141c22bbfa37538cd10cedda16dd0f61ce10a5e53c124b375c1b8a8d5"} Oct 30 06:28:59.532571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:59.532524 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" podStartSLOduration=1.5325089269999999 podStartE2EDuration="1.532508927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:59.531806526 +0000 UTC m=+2148.338328275" watchObservedRunningTime="2026-10-30 06:28:59.532508927 +0000 UTC m=+2148.339030674" Oct 30 06:28:59.850641 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:28:59.850506 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:29:00.500611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:00.500569 4431 generic.go:356] "Generic (PLEG): container finished" podID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerID="1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e" exitCode=0 Oct 30 06:29:00.501331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:00.501288 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" event={"ID":"54aa0ffd-3489-4d17-9171-f56a8664045c","Type":"ContainerDied","Data":"1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e"} Oct 30 06:29:00.501683 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:00.501666 4431 scope.go:122] "RemoveContainer" containerID="1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e" Oct 30 06:29:00.503347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:00.503284 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:29:00.503478 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:00.503399 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:29:00.503478 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:00.503436 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:29:01.509174 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:01.509136 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" event={"ID":"54aa0ffd-3489-4d17-9171-f56a8664045c","Type":"ContainerStarted","Data":"7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819"} Oct 30 06:29:02.383327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:02.382741 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:29:02.390248 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:02.389494 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 06:29:02.390248 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:02.389565 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 06:29:03.384328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:03.383597 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 06:29:03.384328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:03.383673 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 06:29:03.711947 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:29:03.711908 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:29:04.388870 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:04.388516 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 06:29:04.388870 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:04.388577 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 06:29:05.380697 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:05.380656 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 06:29:05.381101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:05.381067 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 06:29:05.948095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:05.946953 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:29:06.382385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:06.381899 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 06:29:06.382385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:06.382000 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 06:29:07.377511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:07.377475 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:29:07.378596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:07.378574 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:29:08.850249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:08.850208 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:29:08.864284 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:08.864250 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:29:09.573064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:09.573035 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:29:12.326137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.326100 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns"] Oct 30 06:29:12.332270 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.332243 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.347940 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.347905 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns"] Oct 30 06:29:12.355139 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.355113 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4f3973\"/\"openshift-service-ca.crt\"" Oct 30 06:29:12.355139 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.355113 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4f3973\"/\"kube-root-ca.crt\"" Oct 30 06:29:12.358774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.358752 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4f3973\"/\"default-dockercfg-bxxnx\"" Oct 30 06:29:12.358970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.358944 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4f3973\"/\"auth-test-model-4f3973-kserve-self-signed-certs\"" Oct 30 06:29:12.510797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.510758 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-home\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.511025 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.510817 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tls-certs\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.511025 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.510977 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tmp-dir\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.511105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.511041 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7k52\" (UniqueName: \"kubernetes.io/projected/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-kube-api-access-h7k52\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.511138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.511120 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-model-cache\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.511239 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.511226 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-dshm\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.614122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.613266 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-model-cache\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.614122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.613423 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-dshm\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.614122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.613889 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-model-cache\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.614699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.614667 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-home\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.614854 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.614759 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tls-certs\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.614943 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.614923 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tmp-dir\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.615044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.615019 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7k52\" (UniqueName: \"kubernetes.io/projected/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-kube-api-access-h7k52\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.615684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.615663 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-home\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.617631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.617606 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tmp-dir\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.628922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.628886 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-dshm\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.630270 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.630239 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tls-certs\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.642538 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.642506 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7k52\" (UniqueName: \"kubernetes.io/projected/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-kube-api-access-h7k52\") pod \"auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.647047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.646999 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:12.957176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:12.957141 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns"] Oct 30 06:29:12.957343 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:29:12.957243 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee10f863_2e1b_4c4a_8d5d_0ab8ce4c9b21.slice/crio-7403e406114d85647df30f3decaccaebe446baaec88b72e0401ac2d2458a07b0 WatchSource:0}: Error finding container 7403e406114d85647df30f3decaccaebe446baaec88b72e0401ac2d2458a07b0: Status 404 returned error can't find the container with id 7403e406114d85647df30f3decaccaebe446baaec88b72e0401ac2d2458a07b0 Oct 30 06:29:13.608316 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:13.608270 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" event={"ID":"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21","Type":"ContainerStarted","Data":"23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40"} Oct 30 06:29:13.608813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:13.608349 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" event={"ID":"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21","Type":"ContainerStarted","Data":"7403e406114d85647df30f3decaccaebe446baaec88b72e0401ac2d2458a07b0"} Oct 30 06:29:13.647579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:13.647531 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:13.649600 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:13.649536 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" podStartSLOduration=1.64952108 podStartE2EDuration="1.64952108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:13.64936349 +0000 UTC m=+2162.455885234" watchObservedRunningTime="2026-10-30 06:29:13.64952108 +0000 UTC m=+2162.456042828" Oct 30 06:29:22.647855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:22.647818 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:22.664786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:22.664755 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:22.679609 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:29:22.679571 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:29:33.702462 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:29:33.702422 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:30:00.181470 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.181373 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d"] Oct 30 06:30:00.186061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.186022 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:00.186061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.186074 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4"] Oct 30 06:30:00.192192 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.192161 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:00.214340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.214294 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d"] Oct 30 06:30:00.218569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.218539 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf"] Oct 30 06:30:00.235222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.232718 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk"] Oct 30 06:30:00.236743 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.235696 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:00.240692 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.240662 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62de82a1-1015-4a3e-be9a-9d401b315f52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d\" (UID: \"62de82a1-1015-4a3e-be9a-9d401b315f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:00.240853 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.240760 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4\" (UID: \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:00.240853 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.240823 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqt8p\" (UniqueName: \"kubernetes.io/projected/62de82a1-1015-4a3e-be9a-9d401b315f52-kube-api-access-xqt8p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d\" (UID: \"62de82a1-1015-4a3e-be9a-9d401b315f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:00.240971 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.240875 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrg29\" (UniqueName: \"kubernetes.io/projected/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-kube-api-access-nrg29\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4\" (UID: \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:00.242724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.242704 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4"] Oct 30 06:30:00.242864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.242743 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf"] Oct 30 06:30:00.242864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.242817 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk"] Oct 30 06:30:00.242942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.242903 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:00.273845 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.273817 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:30:00.342125 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342095 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62de82a1-1015-4a3e-be9a-9d401b315f52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d\" (UID: \"62de82a1-1015-4a3e-be9a-9d401b315f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:00.342332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342267 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4\" (UID: \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:00.342409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342340 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nnlq\" (UniqueName: \"kubernetes.io/projected/d621c82d-ef7d-4db5-a499-d4f4326c4120-kube-api-access-8nnlq\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk\" (UID: \"d621c82d-ef7d-4db5-a499-d4f4326c4120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:00.342409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342374 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqxj\" (UniqueName: \"kubernetes.io/projected/6902e2d6-19fc-440d-8434-d7a00183ea60-kube-api-access-wxqxj\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf\" (UID: \"6902e2d6-19fc-440d-8434-d7a00183ea60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:00.342522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342409 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6902e2d6-19fc-440d-8434-d7a00183ea60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf\" (UID: \"6902e2d6-19fc-440d-8434-d7a00183ea60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:00.342522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342508 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d621c82d-ef7d-4db5-a499-d4f4326c4120-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk\" (UID: \"d621c82d-ef7d-4db5-a499-d4f4326c4120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:00.342684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342659 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqt8p\" (UniqueName: \"kubernetes.io/projected/62de82a1-1015-4a3e-be9a-9d401b315f52-kube-api-access-xqt8p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d\" (UID: \"62de82a1-1015-4a3e-be9a-9d401b315f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:00.342756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342737 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrg29\" (UniqueName: \"kubernetes.io/projected/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-kube-api-access-nrg29\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4\" (UID: \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:00.342919 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342899 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62de82a1-1015-4a3e-be9a-9d401b315f52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d\" (UID: \"62de82a1-1015-4a3e-be9a-9d401b315f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:00.343006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.342960 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4\" (UID: \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:00.365831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.365792 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqt8p\" (UniqueName: \"kubernetes.io/projected/62de82a1-1015-4a3e-be9a-9d401b315f52-kube-api-access-xqt8p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d\" (UID: \"62de82a1-1015-4a3e-be9a-9d401b315f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:00.366003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.365792 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrg29\" (UniqueName: \"kubernetes.io/projected/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-kube-api-access-nrg29\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4\" (UID: \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:00.444407 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.444363 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nnlq\" (UniqueName: \"kubernetes.io/projected/d621c82d-ef7d-4db5-a499-d4f4326c4120-kube-api-access-8nnlq\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk\" (UID: \"d621c82d-ef7d-4db5-a499-d4f4326c4120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:00.444617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.444413 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqxj\" (UniqueName: \"kubernetes.io/projected/6902e2d6-19fc-440d-8434-d7a00183ea60-kube-api-access-wxqxj\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf\" (UID: \"6902e2d6-19fc-440d-8434-d7a00183ea60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:00.444617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.444447 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6902e2d6-19fc-440d-8434-d7a00183ea60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf\" (UID: \"6902e2d6-19fc-440d-8434-d7a00183ea60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:00.444617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.444469 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d621c82d-ef7d-4db5-a499-d4f4326c4120-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk\" (UID: \"d621c82d-ef7d-4db5-a499-d4f4326c4120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:00.445220 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.445199 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d621c82d-ef7d-4db5-a499-d4f4326c4120-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk\" (UID: \"d621c82d-ef7d-4db5-a499-d4f4326c4120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:00.445260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.445208 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6902e2d6-19fc-440d-8434-d7a00183ea60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf\" (UID: \"6902e2d6-19fc-440d-8434-d7a00183ea60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:00.467806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.467768 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nnlq\" (UniqueName: \"kubernetes.io/projected/d621c82d-ef7d-4db5-a499-d4f4326c4120-kube-api-access-8nnlq\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk\" (UID: \"d621c82d-ef7d-4db5-a499-d4f4326c4120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:00.467963 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.467830 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqxj\" (UniqueName: \"kubernetes.io/projected/6902e2d6-19fc-440d-8434-d7a00183ea60-kube-api-access-wxqxj\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf\" (UID: \"6902e2d6-19fc-440d-8434-d7a00183ea60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:00.512379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.512342 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:00.532909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.532872 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:00.557911 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.557869 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:00.566118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.566089 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:00.826116 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.826085 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d"] Oct 30 06:30:00.826796 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:30:00.826763 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62de82a1_1015_4a3e_be9a_9d401b315f52.slice/crio-bb2cec775528ad4fbcc1c69d3e917af48cd8bdadeee194b17dc2df081d7ff509 WatchSource:0}: Error finding container bb2cec775528ad4fbcc1c69d3e917af48cd8bdadeee194b17dc2df081d7ff509: Status 404 returned error can't find the container with id bb2cec775528ad4fbcc1c69d3e917af48cd8bdadeee194b17dc2df081d7ff509 Oct 30 06:30:00.854294 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.854265 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4"] Oct 30 06:30:00.869922 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:30:00.861346 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa652ec_8e3e_474b_8208_b9ee761b9ef9.slice/crio-7d69020481a1b16bbff2e64d966134a01acc8db979080ce63675ff5bd9129fcd WatchSource:0}: Error finding container 7d69020481a1b16bbff2e64d966134a01acc8db979080ce63675ff5bd9129fcd: Status 404 returned error can't find the container with id 7d69020481a1b16bbff2e64d966134a01acc8db979080ce63675ff5bd9129fcd Oct 30 06:30:00.904866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.904834 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk"] Oct 30 06:30:00.905721 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:30:00.905687 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd621c82d_ef7d_4db5_a499_d4f4326c4120.slice/crio-8365017243e8e34ee2f355ff413595aaf58ee2296d2dbde06f04c8a4dd1ebed8 WatchSource:0}: Error finding container 8365017243e8e34ee2f355ff413595aaf58ee2296d2dbde06f04c8a4dd1ebed8: Status 404 returned error can't find the container with id 8365017243e8e34ee2f355ff413595aaf58ee2296d2dbde06f04c8a4dd1ebed8 Oct 30 06:30:00.914609 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.914575 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf"] Oct 30 06:30:00.957826 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.957783 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" event={"ID":"d621c82d-ef7d-4db5-a499-d4f4326c4120","Type":"ContainerStarted","Data":"8365017243e8e34ee2f355ff413595aaf58ee2296d2dbde06f04c8a4dd1ebed8"} Oct 30 06:30:00.960115 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.960081 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" event={"ID":"6902e2d6-19fc-440d-8434-d7a00183ea60","Type":"ContainerStarted","Data":"830d218b50722e01476251280dcb6399785f51915cefb4892b95c9d73a7f83c5"} Oct 30 06:30:00.964406 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.964374 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" event={"ID":"62de82a1-1015-4a3e-be9a-9d401b315f52","Type":"ContainerStarted","Data":"bb2cec775528ad4fbcc1c69d3e917af48cd8bdadeee194b17dc2df081d7ff509"} Oct 30 06:30:00.966151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:00.966126 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" event={"ID":"3aa652ec-8e3e-474b-8208-b9ee761b9ef9","Type":"ContainerStarted","Data":"7d69020481a1b16bbff2e64d966134a01acc8db979080ce63675ff5bd9129fcd"} Oct 30 06:30:01.975046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:01.974998 4431 generic.go:356] "Generic (PLEG): container finished" podID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerID="ef496ba0423c3244b3503bcea304046e648cffc22a7ca10c76fdafb0af47aebd" exitCode=60 Oct 30 06:30:01.975543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:01.975071 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" event={"ID":"6902e2d6-19fc-440d-8434-d7a00183ea60","Type":"ContainerDied","Data":"ef496ba0423c3244b3503bcea304046e648cffc22a7ca10c76fdafb0af47aebd"} Oct 30 06:30:01.975543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:01.975438 4431 scope.go:122] "RemoveContainer" containerID="ef496ba0423c3244b3503bcea304046e648cffc22a7ca10c76fdafb0af47aebd" Oct 30 06:30:01.977177 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:01.977126 4431 generic.go:356] "Generic (PLEG): container finished" podID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerID="b7a27ae3f9bcb871d2815868f4a91747a9d80d7fe9f86cab960d978c01713e93" exitCode=60 Oct 30 06:30:01.977177 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:01.977159 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" event={"ID":"62de82a1-1015-4a3e-be9a-9d401b315f52","Type":"ContainerDied","Data":"b7a27ae3f9bcb871d2815868f4a91747a9d80d7fe9f86cab960d978c01713e93"} Oct 30 06:30:01.977556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:01.977536 4431 scope.go:122] "RemoveContainer" containerID="b7a27ae3f9bcb871d2815868f4a91747a9d80d7fe9f86cab960d978c01713e93" Oct 30 06:30:01.982235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:01.982204 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" event={"ID":"3aa652ec-8e3e-474b-8208-b9ee761b9ef9","Type":"ContainerStarted","Data":"9ba9a5c0520107d15da8215f01862af50efe200ad47e5d2b6af17a0415bdc0a5"} Oct 30 06:30:01.982687 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:01.982669 4431 scope.go:122] "RemoveContainer" containerID="9ba9a5c0520107d15da8215f01862af50efe200ad47e5d2b6af17a0415bdc0a5" Oct 30 06:30:01.984542 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:01.984515 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" event={"ID":"d621c82d-ef7d-4db5-a499-d4f4326c4120","Type":"ContainerStarted","Data":"a9dbfdca6bf00b921d1bbab973ee393f2d7a03cccc22cabcf7a038a64932615b"} Oct 30 06:30:02.098575 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:02.098515 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" podStartSLOduration=2.098501601 podStartE2EDuration="2.098501601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:02.097511754 +0000 UTC m=+2210.904033503" watchObservedRunningTime="2026-10-30 06:30:02.098501601 +0000 UTC m=+2210.905023349" Oct 30 06:30:02.994318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:02.994276 4431 generic.go:356] "Generic (PLEG): container finished" podID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerID="fec20e205ccd3b28f3fa5f1041ccb742af0229fb0f9b99fb3cccbb5d19d6f627" exitCode=60 Oct 30 06:30:02.994765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:02.994336 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" event={"ID":"6902e2d6-19fc-440d-8434-d7a00183ea60","Type":"ContainerDied","Data":"fec20e205ccd3b28f3fa5f1041ccb742af0229fb0f9b99fb3cccbb5d19d6f627"} Oct 30 06:30:02.994765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:02.994404 4431 scope.go:122] "RemoveContainer" containerID="ef496ba0423c3244b3503bcea304046e648cffc22a7ca10c76fdafb0af47aebd" Oct 30 06:30:02.994765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:02.994758 4431 scope.go:122] "RemoveContainer" containerID="fec20e205ccd3b28f3fa5f1041ccb742af0229fb0f9b99fb3cccbb5d19d6f627" Oct 30 06:30:02.995123 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:02.995097 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf_odh-ai-gateway-infra(6902e2d6-19fc-440d-8434-d7a00183ea60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" Oct 30 06:30:02.996762 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:02.996727 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" event={"ID":"62de82a1-1015-4a3e-be9a-9d401b315f52","Type":"ContainerStarted","Data":"8804808728f78bf451a9a549d128863969b518095091a342d0aa5dee2458e949"} Oct 30 06:30:02.998856 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:02.998831 4431 generic.go:356] "Generic (PLEG): container finished" podID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerID="9ba9a5c0520107d15da8215f01862af50efe200ad47e5d2b6af17a0415bdc0a5" exitCode=60 Oct 30 06:30:02.998976 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:02.998911 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" event={"ID":"3aa652ec-8e3e-474b-8208-b9ee761b9ef9","Type":"ContainerDied","Data":"9ba9a5c0520107d15da8215f01862af50efe200ad47e5d2b6af17a0415bdc0a5"} Oct 30 06:30:02.998976 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:02.998965 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" event={"ID":"3aa652ec-8e3e-474b-8208-b9ee761b9ef9","Type":"ContainerStarted","Data":"770d9cce7d02edd2c63af7d2c655e84a02add9a0a205c7beb32521e560189c39"} Oct 30 06:30:03.000810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:03.000788 4431 generic.go:356] "Generic (PLEG): container finished" podID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerID="a9dbfdca6bf00b921d1bbab973ee393f2d7a03cccc22cabcf7a038a64932615b" exitCode=60 Oct 30 06:30:03.000911 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:03.000832 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" event={"ID":"d621c82d-ef7d-4db5-a499-d4f4326c4120","Type":"ContainerDied","Data":"a9dbfdca6bf00b921d1bbab973ee393f2d7a03cccc22cabcf7a038a64932615b"} Oct 30 06:30:03.001096 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:03.001082 4431 scope.go:122] "RemoveContainer" containerID="a9dbfdca6bf00b921d1bbab973ee393f2d7a03cccc22cabcf7a038a64932615b" Oct 30 06:30:03.075925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:03.075871 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" podStartSLOduration=3.075853297 podStartE2EDuration="3.075853297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:03.072528339 +0000 UTC m=+2211.879050088" watchObservedRunningTime="2026-10-30 06:30:03.075853297 +0000 UTC m=+2211.882375074" Oct 30 06:30:03.153514 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:03.153453 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" podStartSLOduration=3.15343513 podStartE2EDuration="3.15343513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:03.151147544 +0000 UTC m=+2211.957669292" watchObservedRunningTime="2026-10-30 06:30:03.15343513 +0000 UTC m=+2211.959956868" Oct 30 06:30:03.771757 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:03.771716 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:30:04.010615 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.010587 4431 scope.go:122] "RemoveContainer" containerID="fec20e205ccd3b28f3fa5f1041ccb742af0229fb0f9b99fb3cccbb5d19d6f627" Oct 30 06:30:04.011057 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:04.010935 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf_odh-ai-gateway-infra(6902e2d6-19fc-440d-8434-d7a00183ea60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" Oct 30 06:30:04.012367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.012341 4431 generic.go:356] "Generic (PLEG): container finished" podID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerID="8804808728f78bf451a9a549d128863969b518095091a342d0aa5dee2458e949" exitCode=60 Oct 30 06:30:04.012473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.012436 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" event={"ID":"62de82a1-1015-4a3e-be9a-9d401b315f52","Type":"ContainerDied","Data":"8804808728f78bf451a9a549d128863969b518095091a342d0aa5dee2458e949"} Oct 30 06:30:04.012546 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.012516 4431 scope.go:122] "RemoveContainer" containerID="b7a27ae3f9bcb871d2815868f4a91747a9d80d7fe9f86cab960d978c01713e93" Oct 30 06:30:04.012829 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.012808 4431 scope.go:122] "RemoveContainer" containerID="8804808728f78bf451a9a549d128863969b518095091a342d0aa5dee2458e949" Oct 30 06:30:04.013185 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:04.013160 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d_odh-ai-gateway-infra(62de82a1-1015-4a3e-be9a-9d401b315f52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" Oct 30 06:30:04.014566 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.014540 4431 generic.go:356] "Generic (PLEG): container finished" podID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerID="770d9cce7d02edd2c63af7d2c655e84a02add9a0a205c7beb32521e560189c39" exitCode=60 Oct 30 06:30:04.014669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.014621 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" event={"ID":"3aa652ec-8e3e-474b-8208-b9ee761b9ef9","Type":"ContainerDied","Data":"770d9cce7d02edd2c63af7d2c655e84a02add9a0a205c7beb32521e560189c39"} Oct 30 06:30:04.014914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.014897 4431 scope.go:122] "RemoveContainer" containerID="770d9cce7d02edd2c63af7d2c655e84a02add9a0a205c7beb32521e560189c39" Oct 30 06:30:04.015283 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:04.015215 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4_odh-ai-gateway-infra(3aa652ec-8e3e-474b-8208-b9ee761b9ef9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" Oct 30 06:30:04.018496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.018469 4431 generic.go:356] "Generic (PLEG): container finished" podID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerID="fbe4530cf3263675254aa9cce64c5b17491469d96a59ac171122b5050474888d" exitCode=60 Oct 30 06:30:04.018596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.018547 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" event={"ID":"d621c82d-ef7d-4db5-a499-d4f4326c4120","Type":"ContainerDied","Data":"fbe4530cf3263675254aa9cce64c5b17491469d96a59ac171122b5050474888d"} Oct 30 06:30:04.018879 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.018864 4431 scope.go:122] "RemoveContainer" containerID="fbe4530cf3263675254aa9cce64c5b17491469d96a59ac171122b5050474888d" Oct 30 06:30:04.019218 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:04.019198 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk_odh-ai-gateway-infra(d621c82d-ef7d-4db5-a499-d4f4326c4120)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" Oct 30 06:30:04.028103 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.028087 4431 scope.go:122] "RemoveContainer" containerID="9ba9a5c0520107d15da8215f01862af50efe200ad47e5d2b6af17a0415bdc0a5" Oct 30 06:30:04.041401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:04.041374 4431 scope.go:122] "RemoveContainer" containerID="a9dbfdca6bf00b921d1bbab973ee393f2d7a03cccc22cabcf7a038a64932615b" Oct 30 06:30:05.027825 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:05.027794 4431 scope.go:122] "RemoveContainer" containerID="fbe4530cf3263675254aa9cce64c5b17491469d96a59ac171122b5050474888d" Oct 30 06:30:05.028497 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:05.028185 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk_odh-ai-gateway-infra(d621c82d-ef7d-4db5-a499-d4f4326c4120)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" Oct 30 06:30:05.030675 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:05.030648 4431 scope.go:122] "RemoveContainer" containerID="8804808728f78bf451a9a549d128863969b518095091a342d0aa5dee2458e949" Oct 30 06:30:05.031050 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:05.031025 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d_odh-ai-gateway-infra(62de82a1-1015-4a3e-be9a-9d401b315f52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" Oct 30 06:30:05.032457 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:05.032437 4431 scope.go:122] "RemoveContainer" containerID="770d9cce7d02edd2c63af7d2c655e84a02add9a0a205c7beb32521e560189c39" Oct 30 06:30:05.032749 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:05.032729 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4_odh-ai-gateway-infra(3aa652ec-8e3e-474b-8208-b9ee761b9ef9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" Oct 30 06:30:15.330936 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:15.330899 4431 scope.go:122] "RemoveContainer" containerID="fbe4530cf3263675254aa9cce64c5b17491469d96a59ac171122b5050474888d" Oct 30 06:30:15.331389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:15.331090 4431 scope.go:122] "RemoveContainer" containerID="770d9cce7d02edd2c63af7d2c655e84a02add9a0a205c7beb32521e560189c39" Oct 30 06:30:16.103257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:16.103218 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" event={"ID":"3aa652ec-8e3e-474b-8208-b9ee761b9ef9","Type":"ContainerStarted","Data":"10c0e633e19e5662318c6d31265956b2295b9515db669c5ecdc2316a25df902f"} Oct 30 06:30:16.105367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:16.105319 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" event={"ID":"d621c82d-ef7d-4db5-a499-d4f4326c4120","Type":"ContainerStarted","Data":"e86e9140207b369ab59f66421bcec42625404d76e67d78b39365bed87dec8b54"} Oct 30 06:30:16.331423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:16.331390 4431 scope.go:122] "RemoveContainer" containerID="8804808728f78bf451a9a549d128863969b518095091a342d0aa5dee2458e949" Oct 30 06:30:16.331831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:16.331558 4431 scope.go:122] "RemoveContainer" containerID="fec20e205ccd3b28f3fa5f1041ccb742af0229fb0f9b99fb3cccbb5d19d6f627" Oct 30 06:30:17.114189 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.114150 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" event={"ID":"6902e2d6-19fc-440d-8434-d7a00183ea60","Type":"ContainerStarted","Data":"ccc406923162b66788da69ea9925fca9e49b7250543306a6e86c9869adadf93f"} Oct 30 06:30:17.116399 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.116364 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" event={"ID":"62de82a1-1015-4a3e-be9a-9d401b315f52","Type":"ContainerStarted","Data":"3de8fbe65c6374d700f0c260408ce06df63bb7139891e6714f7845c2f723c153"} Oct 30 06:30:17.118224 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.118200 4431 generic.go:356] "Generic (PLEG): container finished" podID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerID="10c0e633e19e5662318c6d31265956b2295b9515db669c5ecdc2316a25df902f" exitCode=60 Oct 30 06:30:17.118379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.118271 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" event={"ID":"3aa652ec-8e3e-474b-8208-b9ee761b9ef9","Type":"ContainerDied","Data":"10c0e633e19e5662318c6d31265956b2295b9515db669c5ecdc2316a25df902f"} Oct 30 06:30:17.118379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.118336 4431 scope.go:122] "RemoveContainer" containerID="770d9cce7d02edd2c63af7d2c655e84a02add9a0a205c7beb32521e560189c39" Oct 30 06:30:17.118732 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.118713 4431 scope.go:122] "RemoveContainer" containerID="10c0e633e19e5662318c6d31265956b2295b9515db669c5ecdc2316a25df902f" Oct 30 06:30:17.119051 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:17.119022 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4_odh-ai-gateway-infra(3aa652ec-8e3e-474b-8208-b9ee761b9ef9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" Oct 30 06:30:17.120381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.120357 4431 generic.go:356] "Generic (PLEG): container finished" podID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerID="e86e9140207b369ab59f66421bcec42625404d76e67d78b39365bed87dec8b54" exitCode=60 Oct 30 06:30:17.120467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.120400 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" event={"ID":"d621c82d-ef7d-4db5-a499-d4f4326c4120","Type":"ContainerDied","Data":"e86e9140207b369ab59f66421bcec42625404d76e67d78b39365bed87dec8b54"} Oct 30 06:30:17.120710 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.120698 4431 scope.go:122] "RemoveContainer" containerID="e86e9140207b369ab59f66421bcec42625404d76e67d78b39365bed87dec8b54" Oct 30 06:30:17.120947 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:17.120931 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk_odh-ai-gateway-infra(d621c82d-ef7d-4db5-a499-d4f4326c4120)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" Oct 30 06:30:17.130900 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.130875 4431 scope.go:122] "RemoveContainer" containerID="fbe4530cf3263675254aa9cce64c5b17491469d96a59ac171122b5050474888d" Oct 30 06:30:17.154936 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.154902 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4"] Oct 30 06:30:17.159259 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.159213 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" podStartSLOduration=17.15919889 podStartE2EDuration="17.15919889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:17.15710882 +0000 UTC m=+2225.963630571" watchObservedRunningTime="2026-10-30 06:30:17.15919889 +0000 UTC m=+2225.965720638" Oct 30 06:30:17.221536 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:17.221492 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk"] Oct 30 06:30:18.129969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.129929 4431 generic.go:356] "Generic (PLEG): container finished" podID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerID="ccc406923162b66788da69ea9925fca9e49b7250543306a6e86c9869adadf93f" exitCode=60 Oct 30 06:30:18.130447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.130004 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" event={"ID":"6902e2d6-19fc-440d-8434-d7a00183ea60","Type":"ContainerDied","Data":"ccc406923162b66788da69ea9925fca9e49b7250543306a6e86c9869adadf93f"} Oct 30 06:30:18.130447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.130119 4431 scope.go:122] "RemoveContainer" containerID="fec20e205ccd3b28f3fa5f1041ccb742af0229fb0f9b99fb3cccbb5d19d6f627" Oct 30 06:30:18.130571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.130553 4431 scope.go:122] "RemoveContainer" containerID="ccc406923162b66788da69ea9925fca9e49b7250543306a6e86c9869adadf93f" Oct 30 06:30:18.130965 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:18.130940 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf_odh-ai-gateway-infra(6902e2d6-19fc-440d-8434-d7a00183ea60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" Oct 30 06:30:18.132246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.132220 4431 generic.go:356] "Generic (PLEG): container finished" podID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerID="3de8fbe65c6374d700f0c260408ce06df63bb7139891e6714f7845c2f723c153" exitCode=60 Oct 30 06:30:18.132390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.132295 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" event={"ID":"62de82a1-1015-4a3e-be9a-9d401b315f52","Type":"ContainerDied","Data":"3de8fbe65c6374d700f0c260408ce06df63bb7139891e6714f7845c2f723c153"} Oct 30 06:30:18.132731 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.132712 4431 scope.go:122] "RemoveContainer" containerID="3de8fbe65c6374d700f0c260408ce06df63bb7139891e6714f7845c2f723c153" Oct 30 06:30:18.133090 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:18.133058 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d_odh-ai-gateway-infra(62de82a1-1015-4a3e-be9a-9d401b315f52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" Oct 30 06:30:18.144656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.144629 4431 scope.go:122] "RemoveContainer" containerID="8804808728f78bf451a9a549d128863969b518095091a342d0aa5dee2458e949" Oct 30 06:30:18.170511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.170474 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf"] Oct 30 06:30:18.265348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.265275 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d"] Oct 30 06:30:18.318281 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.318239 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:18.355392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.355368 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:18.384980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.384870 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d621c82d-ef7d-4db5-a499-d4f4326c4120-kube-api-access-8nnlq\" (UniqueName: \"kubernetes.io/projected/d621c82d-ef7d-4db5-a499-d4f4326c4120-kube-api-access-8nnlq\") pod \"d621c82d-ef7d-4db5-a499-d4f4326c4120\" (UID: \"d621c82d-ef7d-4db5-a499-d4f4326c4120\") " Oct 30 06:30:18.384980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.384925 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d621c82d-ef7d-4db5-a499-d4f4326c4120-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d621c82d-ef7d-4db5-a499-d4f4326c4120-maas-api-service-ca\") pod \"d621c82d-ef7d-4db5-a499-d4f4326c4120\" (UID: \"d621c82d-ef7d-4db5-a499-d4f4326c4120\") " Oct 30 06:30:18.385462 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.385430 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d621c82d-ef7d-4db5-a499-d4f4326c4120-maas-api-service-ca" pod "d621c82d-ef7d-4db5-a499-d4f4326c4120" (UID: "d621c82d-ef7d-4db5-a499-d4f4326c4120"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:18.387772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.387744 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d621c82d-ef7d-4db5-a499-d4f4326c4120-kube-api-access-8nnlq" pod "d621c82d-ef7d-4db5-a499-d4f4326c4120" (UID: "d621c82d-ef7d-4db5-a499-d4f4326c4120"). InnerVolumeSpecName "kube-api-access-8nnlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:18.486559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.486524 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-maas-api-service-ca\") pod \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\" (UID: \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\") " Oct 30 06:30:18.486780 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.486657 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-kube-api-access-nrg29\" (UniqueName: \"kubernetes.io/projected/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-kube-api-access-nrg29\") pod \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\" (UID: \"3aa652ec-8e3e-474b-8208-b9ee761b9ef9\") " Oct 30 06:30:18.486901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.486871 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-maas-api-service-ca" pod "3aa652ec-8e3e-474b-8208-b9ee761b9ef9" (UID: "3aa652ec-8e3e-474b-8208-b9ee761b9ef9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:18.487219 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.487199 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:30:18.487351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.487223 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nnlq\" (UniqueName: \"kubernetes.io/projected/d621c82d-ef7d-4db5-a499-d4f4326c4120-kube-api-access-8nnlq\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:30:18.487351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.487236 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d621c82d-ef7d-4db5-a499-d4f4326c4120-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:30:18.488824 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.488797 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-kube-api-access-nrg29" pod "3aa652ec-8e3e-474b-8208-b9ee761b9ef9" (UID: "3aa652ec-8e3e-474b-8208-b9ee761b9ef9"). InnerVolumeSpecName "kube-api-access-nrg29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:18.588399 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:18.588357 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrg29\" (UniqueName: \"kubernetes.io/projected/3aa652ec-8e3e-474b-8208-b9ee761b9ef9-kube-api-access-nrg29\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.153476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.153444 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" Oct 30 06:30:19.153909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.153502 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4" event={"ID":"3aa652ec-8e3e-474b-8208-b9ee761b9ef9","Type":"ContainerDied","Data":"7d69020481a1b16bbff2e64d966134a01acc8db979080ce63675ff5bd9129fcd"} Oct 30 06:30:19.153909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.153572 4431 scope.go:122] "RemoveContainer" containerID="10c0e633e19e5662318c6d31265956b2295b9515db669c5ecdc2316a25df902f" Oct 30 06:30:19.155809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.155775 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" event={"ID":"d621c82d-ef7d-4db5-a499-d4f4326c4120","Type":"ContainerDied","Data":"8365017243e8e34ee2f355ff413595aaf58ee2296d2dbde06f04c8a4dd1ebed8"} Oct 30 06:30:19.155906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.155895 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk" Oct 30 06:30:19.169062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.168756 4431 scope.go:122] "RemoveContainer" containerID="e86e9140207b369ab59f66421bcec42625404d76e67d78b39365bed87dec8b54" Oct 30 06:30:19.232573 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.225156 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk"] Oct 30 06:30:19.232573 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.231115 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-b9ff93-29889030-p8szk"] Oct 30 06:30:19.300573 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.300537 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4"] Oct 30 06:30:19.306654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.306620 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889030-sqrf4"] Oct 30 06:30:19.350934 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.350896 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:19.352250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.352219 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" path="/var/lib/kubelet/pods/3aa652ec-8e3e-474b-8208-b9ee761b9ef9/volumes" Oct 30 06:30:19.352917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.352896 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" path="/var/lib/kubelet/pods/d621c82d-ef7d-4db5-a499-d4f4326c4120/volumes" Oct 30 06:30:19.366175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.366153 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:19.509918 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.509886 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62de82a1-1015-4a3e-be9a-9d401b315f52-kube-api-access-xqt8p\" (UniqueName: \"kubernetes.io/projected/62de82a1-1015-4a3e-be9a-9d401b315f52-kube-api-access-xqt8p\") pod \"62de82a1-1015-4a3e-be9a-9d401b315f52\" (UID: \"62de82a1-1015-4a3e-be9a-9d401b315f52\") " Oct 30 06:30:19.510113 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.509967 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6902e2d6-19fc-440d-8434-d7a00183ea60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6902e2d6-19fc-440d-8434-d7a00183ea60-maas-api-service-ca\") pod \"6902e2d6-19fc-440d-8434-d7a00183ea60\" (UID: \"6902e2d6-19fc-440d-8434-d7a00183ea60\") " Oct 30 06:30:19.510113 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.510089 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62de82a1-1015-4a3e-be9a-9d401b315f52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62de82a1-1015-4a3e-be9a-9d401b315f52-maas-api-service-ca\") pod \"62de82a1-1015-4a3e-be9a-9d401b315f52\" (UID: \"62de82a1-1015-4a3e-be9a-9d401b315f52\") " Oct 30 06:30:19.510229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.510113 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6902e2d6-19fc-440d-8434-d7a00183ea60-kube-api-access-wxqxj\" (UniqueName: \"kubernetes.io/projected/6902e2d6-19fc-440d-8434-d7a00183ea60-kube-api-access-wxqxj\") pod \"6902e2d6-19fc-440d-8434-d7a00183ea60\" (UID: \"6902e2d6-19fc-440d-8434-d7a00183ea60\") " Oct 30 06:30:19.510460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.510427 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6902e2d6-19fc-440d-8434-d7a00183ea60-maas-api-service-ca" pod "6902e2d6-19fc-440d-8434-d7a00183ea60" (UID: "6902e2d6-19fc-440d-8434-d7a00183ea60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:19.510631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.510488 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62de82a1-1015-4a3e-be9a-9d401b315f52-maas-api-service-ca" pod "62de82a1-1015-4a3e-be9a-9d401b315f52" (UID: "62de82a1-1015-4a3e-be9a-9d401b315f52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:19.511140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.510971 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62de82a1-1015-4a3e-be9a-9d401b315f52-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.511140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.510999 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6902e2d6-19fc-440d-8434-d7a00183ea60-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.512377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.512349 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62de82a1-1015-4a3e-be9a-9d401b315f52-kube-api-access-xqt8p" pod "62de82a1-1015-4a3e-be9a-9d401b315f52" (UID: "62de82a1-1015-4a3e-be9a-9d401b315f52"). InnerVolumeSpecName "kube-api-access-xqt8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:19.512521 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.512497 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6902e2d6-19fc-440d-8434-d7a00183ea60-kube-api-access-wxqxj" pod "6902e2d6-19fc-440d-8434-d7a00183ea60" (UID: "6902e2d6-19fc-440d-8434-d7a00183ea60"). InnerVolumeSpecName "kube-api-access-wxqxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:19.612168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.612124 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxqxj\" (UniqueName: \"kubernetes.io/projected/6902e2d6-19fc-440d-8434-d7a00183ea60-kube-api-access-wxqxj\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.612168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:19.612161 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqt8p\" (UniqueName: \"kubernetes.io/projected/62de82a1-1015-4a3e-be9a-9d401b315f52-kube-api-access-xqt8p\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:30:20.168334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.168261 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" Oct 30 06:30:20.169160 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.168656 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf" event={"ID":"6902e2d6-19fc-440d-8434-d7a00183ea60","Type":"ContainerDied","Data":"830d218b50722e01476251280dcb6399785f51915cefb4892b95c9d73a7f83c5"} Oct 30 06:30:20.169288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.169249 4431 scope.go:122] "RemoveContainer" containerID="ccc406923162b66788da69ea9925fca9e49b7250543306a6e86c9869adadf93f" Oct 30 06:30:20.171528 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.171436 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" event={"ID":"62de82a1-1015-4a3e-be9a-9d401b315f52","Type":"ContainerDied","Data":"bb2cec775528ad4fbcc1c69d3e917af48cd8bdadeee194b17dc2df081d7ff509"} Oct 30 06:30:20.171800 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.171780 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d" Oct 30 06:30:20.186072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.186051 4431 scope.go:122] "RemoveContainer" containerID="3de8fbe65c6374d700f0c260408ce06df63bb7139891e6714f7845c2f723c153" Oct 30 06:30:20.238054 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.238017 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf"] Oct 30 06:30:20.247766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.247722 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889030-5p6jf"] Oct 30 06:30:20.279491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.279453 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d"] Oct 30 06:30:20.285788 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:20.285758 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fa5a69-29889030-gfd4d"] Oct 30 06:30:21.338660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:21.338627 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" path="/var/lib/kubelet/pods/62de82a1-1015-4a3e-be9a-9d401b315f52/volumes" Oct 30 06:30:21.339067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:30:21.339058 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" path="/var/lib/kubelet/pods/6902e2d6-19fc-440d-8434-d7a00183ea60/volumes" Oct 30 06:30:33.701846 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:30:33.701810 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:31:03.703677 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:31:03.703640 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:31:26.072115 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:26.072078 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj"] Oct 30 06:31:26.072603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:26.072356 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" containerID="cri-o://fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44" gracePeriod=30 Oct 30 06:31:31.346012 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.345936 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:31:31.401934 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.401448 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-token\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-token\") pod \"82642864-9423-44f0-b32d-4de4b1cf3e2c\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " Oct 30 06:31:31.402260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.401989 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-kube-api-access-rq8r6\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-kube-api-access-rq8r6\") pod \"82642864-9423-44f0-b32d-4de4b1cf3e2c\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " Oct 30 06:31:31.402260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.402072 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-socket\") pod \"82642864-9423-44f0-b32d-4de4b1cf3e2c\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " Oct 30 06:31:31.402260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.402155 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-data\") pod \"82642864-9423-44f0-b32d-4de4b1cf3e2c\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " Oct 30 06:31:31.402260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.402193 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-podinfo\") pod \"82642864-9423-44f0-b32d-4de4b1cf3e2c\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " Oct 30 06:31:31.402260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.402223 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-envoy\") pod \"82642864-9423-44f0-b32d-4de4b1cf3e2c\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " Oct 30 06:31:31.402562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.402285 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-certs\") pod \"82642864-9423-44f0-b32d-4de4b1cf3e2c\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " Oct 30 06:31:31.402562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.402351 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82642864-9423-44f0-b32d-4de4b1cf3e2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82642864-9423-44f0-b32d-4de4b1cf3e2c-istiod-ca-cert\") pod \"82642864-9423-44f0-b32d-4de4b1cf3e2c\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " Oct 30 06:31:31.402562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.402419 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-credential-socket\") pod \"82642864-9423-44f0-b32d-4de4b1cf3e2c\" (UID: \"82642864-9423-44f0-b32d-4de4b1cf3e2c\") " Oct 30 06:31:31.402714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.402561 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-data" pod "82642864-9423-44f0-b32d-4de4b1cf3e2c" (UID: "82642864-9423-44f0-b32d-4de4b1cf3e2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:31.404343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.403491 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-socket" pod "82642864-9423-44f0-b32d-4de4b1cf3e2c" (UID: "82642864-9423-44f0-b32d-4de4b1cf3e2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:31.404343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.403839 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.404343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.403862 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.404343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.404295 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-certs" pod "82642864-9423-44f0-b32d-4de4b1cf3e2c" (UID: "82642864-9423-44f0-b32d-4de4b1cf3e2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:31.404765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.404739 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82642864-9423-44f0-b32d-4de4b1cf3e2c-istiod-ca-cert" pod "82642864-9423-44f0-b32d-4de4b1cf3e2c" (UID: "82642864-9423-44f0-b32d-4de4b1cf3e2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:31.405324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.405119 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-credential-socket" pod "82642864-9423-44f0-b32d-4de4b1cf3e2c" (UID: "82642864-9423-44f0-b32d-4de4b1cf3e2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:31.406456 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.406428 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-token" pod "82642864-9423-44f0-b32d-4de4b1cf3e2c" (UID: "82642864-9423-44f0-b32d-4de4b1cf3e2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:31.407043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.407015 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-kube-api-access-rq8r6" pod "82642864-9423-44f0-b32d-4de4b1cf3e2c" (UID: "82642864-9423-44f0-b32d-4de4b1cf3e2c"). InnerVolumeSpecName "kube-api-access-rq8r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:31.407192 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.407052 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-podinfo" pod "82642864-9423-44f0-b32d-4de4b1cf3e2c" (UID: "82642864-9423-44f0-b32d-4de4b1cf3e2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:31:31.407192 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.407118 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-envoy" pod "82642864-9423-44f0-b32d-4de4b1cf3e2c" (UID: "82642864-9423-44f0-b32d-4de4b1cf3e2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:31.505241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.505198 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82642864-9423-44f0-b32d-4de4b1cf3e2c-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.505241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.505234 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.505241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.505246 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.505549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.505255 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq8r6\" (UniqueName: \"kubernetes.io/projected/82642864-9423-44f0-b32d-4de4b1cf3e2c-kube-api-access-rq8r6\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.505549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.505266 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.505549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.505274 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.505549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.505282 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82642864-9423-44f0-b32d-4de4b1cf3e2c-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.708272 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.708238 4431 generic.go:356] "Generic (PLEG): container finished" podID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerID="fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44" exitCode=0 Oct 30 06:31:31.708479 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.708294 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" Oct 30 06:31:31.708479 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.708336 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" event={"ID":"82642864-9423-44f0-b32d-4de4b1cf3e2c","Type":"ContainerDied","Data":"fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44"} Oct 30 06:31:31.708479 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.708413 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj" event={"ID":"82642864-9423-44f0-b32d-4de4b1cf3e2c","Type":"ContainerDied","Data":"13420930935e677a06dfdff79658fabd8125c32cb173400a39ec2d9f7a388fc8"} Oct 30 06:31:31.708479 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.708447 4431 scope.go:122] "RemoveContainer" containerID="fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44" Oct 30 06:31:31.719596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.719569 4431 scope.go:122] "RemoveContainer" containerID="fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44" Oct 30 06:31:31.719901 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:31:31.719881 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44\": container with ID starting with fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44 not found: ID does not exist" containerID="fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44" Oct 30 06:31:31.719958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.719912 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44"} err="failed to get container status \"fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44\": rpc error: code = NotFound desc = could not find container \"fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44\": container with ID starting with fc67bc05de141fe64a0021dfb8dd5efabd1e0c2f444f564b08933d03b34f5b44 not found: ID does not exist" Oct 30 06:31:31.771202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.771170 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj"] Oct 30 06:31:31.786549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:31.786513 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-fkfdj"] Oct 30 06:31:32.327648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.327603 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk"] Oct 30 06:31:32.328480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328455 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerName="cleanup" Oct 30 06:31:32.328480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328477 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328491 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328499 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328514 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328522 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328536 4431 cpu_manager.go:414] "containerMap: removing container" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328544 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328556 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328563 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328592 4431 cpu_manager.go:414] "containerMap: removing container" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328599 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328610 4431 cpu_manager.go:414] "containerMap: removing container" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328617 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328626 4431 cpu_manager.go:414] "containerMap: removing container" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328633 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328662 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328668 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328676 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerName="cleanup" Oct 30 06:31:32.328682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328683 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328834 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328846 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328857 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328867 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328874 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="62de82a1-1015-4a3e-be9a-9d401b315f52" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328884 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328904 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" containerName="istio-proxy" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328913 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328923 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328933 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerName="cleanup" Oct 30 06:31:32.329630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.328942 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerName="cleanup" Oct 30 06:31:32.338655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.333990 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.356371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.356342 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk"] Oct 30 06:31:32.363841 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.363810 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b8ca49-openshift-default-dockercfg-7dxf8\"" Oct 30 06:31:32.423641 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.423602 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-socket\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.423835 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.423658 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-certs\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.423835 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.423744 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istiod-ca-cert\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.423835 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.423813 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-credential-socket\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.423982 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.423888 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-podinfo\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.424104 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.424078 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-data\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.424147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.424124 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-token\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.424195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.424162 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xdtf\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-kube-api-access-4xdtf\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.424254 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.424236 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-envoy\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.527235 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-data\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.527319 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-token\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.527356 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xdtf\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-kube-api-access-4xdtf\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.527410 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-envoy\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.527495 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-socket\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.527542 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-certs\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.527565 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istiod-ca-cert\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.527637 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-credential-socket\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.527734 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-podinfo\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.529214 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-data\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.529708 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-certs\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.529969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.529955 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-socket\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.536958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.536921 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-envoy\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.544377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.537469 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-credential-socket\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.544622 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.539800 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istiod-ca-cert\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.544727 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.541403 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-podinfo\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.553217 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.553185 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xdtf\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-kube-api-access-4xdtf\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.584950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.584876 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-token\") pod \"e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.651557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.651512 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:32.976597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.976562 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk"] Oct 30 06:31:32.976829 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:31:32.976806 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db731cb_7cbb_4aef_9ef2_820c21f6317d.slice/crio-8fb77764a51c738e123e1a472ce7a5e5ff6eef91afbef5ed44cec4ce8bf0737b WatchSource:0}: Error finding container 8fb77764a51c738e123e1a472ce7a5e5ff6eef91afbef5ed44cec4ce8bf0737b: Status 404 returned error can't find the container with id 8fb77764a51c738e123e1a472ce7a5e5ff6eef91afbef5ed44cec4ce8bf0737b Oct 30 06:31:32.979465 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.979432 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:31:32.979534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.979508 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:31:32.979568 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:32.979538 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:31:33.340912 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:33.340820 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82642864-9423-44f0-b32d-4de4b1cf3e2c" path="/var/lib/kubelet/pods/82642864-9423-44f0-b32d-4de4b1cf3e2c/volumes" Oct 30 06:31:33.703235 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:31:33.703196 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:31:33.728651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:33.728607 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" event={"ID":"1db731cb-7cbb-4aef-9ef2-820c21f6317d","Type":"ContainerStarted","Data":"5d8ad6d08592b2fd578051ccb77223c3bde9552ecc1234885b937ee9abb7ac14"} Oct 30 06:31:33.728909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:33.728675 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" event={"ID":"1db731cb-7cbb-4aef-9ef2-820c21f6317d","Type":"ContainerStarted","Data":"8fb77764a51c738e123e1a472ce7a5e5ff6eef91afbef5ed44cec4ce8bf0737b"} Oct 30 06:31:33.798805 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:33.798746 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" podStartSLOduration=1.798731678 podStartE2EDuration="1.798731678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:33.784911258 +0000 UTC m=+2302.591433003" watchObservedRunningTime="2026-10-30 06:31:33.798731678 +0000 UTC m=+2302.605253426" Oct 30 06:31:34.651690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:34.651646 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:34.657945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:34.657918 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:34.734688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:34.734658 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:34.735906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:34.735880 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:31:39.865970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.865933 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2"] Oct 30 06:31:39.866586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.866569 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerName="cleanup" Oct 30 06:31:39.866642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.866586 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d621c82d-ef7d-4db5-a499-d4f4326c4120" containerName="cleanup" Oct 30 06:31:39.866642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.866597 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerName="cleanup" Oct 30 06:31:39.866642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.866602 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerName="cleanup" Oct 30 06:31:39.866642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.866615 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerName="cleanup" Oct 30 06:31:39.866642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.866620 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerName="cleanup" Oct 30 06:31:39.866796 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.866724 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa652ec-8e3e-474b-8208-b9ee761b9ef9" containerName="cleanup" Oct 30 06:31:39.866796 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.866735 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6902e2d6-19fc-440d-8434-d7a00183ea60" containerName="cleanup" Oct 30 06:31:39.870174 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.870157 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:31:39.888982 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.888951 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2"] Oct 30 06:31:39.897161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.897129 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2d2923\"" Oct 30 06:31:39.930850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.930812 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9edcc58-67e7-4486-a23d-6893a284a2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2\" (UID: \"a9edcc58-67e7-4486-a23d-6893a284a2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:31:39.931031 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:39.930887 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78kh\" (UniqueName: \"kubernetes.io/projected/a9edcc58-67e7-4486-a23d-6893a284a2fd-kube-api-access-r78kh\") pod \"payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2\" (UID: \"a9edcc58-67e7-4486-a23d-6893a284a2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:31:40.036325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.034278 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78kh\" (UniqueName: \"kubernetes.io/projected/a9edcc58-67e7-4486-a23d-6893a284a2fd-kube-api-access-r78kh\") pod \"payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2\" (UID: \"a9edcc58-67e7-4486-a23d-6893a284a2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:31:40.036325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.034614 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9edcc58-67e7-4486-a23d-6893a284a2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2\" (UID: \"a9edcc58-67e7-4486-a23d-6893a284a2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:31:40.036325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.035619 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9edcc58-67e7-4486-a23d-6893a284a2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2\" (UID: \"a9edcc58-67e7-4486-a23d-6893a284a2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:31:40.081335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.075329 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78kh\" (UniqueName: \"kubernetes.io/projected/a9edcc58-67e7-4486-a23d-6893a284a2fd-kube-api-access-r78kh\") pod \"payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2\" (UID: \"a9edcc58-67e7-4486-a23d-6893a284a2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:31:40.185628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.185530 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:31:40.490492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.490458 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2"] Oct 30 06:31:40.491066 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:31:40.491039 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9edcc58_67e7_4486_a23d_6893a284a2fd.slice/crio-4935b43bda4ce14d513dc78cd747dbab4c8ea6c61bb1d90fa7feb8d0d0240ead WatchSource:0}: Error finding container 4935b43bda4ce14d513dc78cd747dbab4c8ea6c61bb1d90fa7feb8d0d0240ead: Status 404 returned error can't find the container with id 4935b43bda4ce14d513dc78cd747dbab4c8ea6c61bb1d90fa7feb8d0d0240ead Oct 30 06:31:40.797625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.797524 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" event={"ID":"a9edcc58-67e7-4486-a23d-6893a284a2fd","Type":"ContainerStarted","Data":"eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412"} Oct 30 06:31:40.797625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.797578 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" event={"ID":"a9edcc58-67e7-4486-a23d-6893a284a2fd","Type":"ContainerStarted","Data":"4935b43bda4ce14d513dc78cd747dbab4c8ea6c61bb1d90fa7feb8d0d0240ead"} Oct 30 06:31:40.797874 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.797655 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:31:40.840981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:40.840927 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" podStartSLOduration=1.840912565 podStartE2EDuration="1.840912565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:40.838214245 +0000 UTC m=+2309.644735994" watchObservedRunningTime="2026-10-30 06:31:40.840912565 +0000 UTC m=+2309.647434353" Oct 30 06:31:51.807013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:31:51.806982 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:32:03.740494 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:32:03.740450 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:32:32.166607 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:32.166508 4431 generic.go:356] "Generic (PLEG): container finished" podID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerID="5d8ad6d08592b2fd578051ccb77223c3bde9552ecc1234885b937ee9abb7ac14" exitCode=0 Oct 30 06:32:32.166607 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:32.166585 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" event={"ID":"1db731cb-7cbb-4aef-9ef2-820c21f6317d","Type":"ContainerDied","Data":"5d8ad6d08592b2fd578051ccb77223c3bde9552ecc1234885b937ee9abb7ac14"} Oct 30 06:32:32.167122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:32.167077 4431 scope.go:122] "RemoveContainer" containerID="5d8ad6d08592b2fd578051ccb77223c3bde9552ecc1234885b937ee9abb7ac14" Oct 30 06:32:32.168905 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:32.168875 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:32:32.168976 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:32.168950 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:32:32.169019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:32.168991 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:32:33.189713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:33.189677 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" event={"ID":"1db731cb-7cbb-4aef-9ef2-820c21f6317d","Type":"ContainerStarted","Data":"7a3840710cab03c3d9d7ee151d3dafbe3b8689bd7c3ffeca3f5a9e13c1cfc1c4"} Oct 30 06:32:33.653095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:33.653054 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:32:33.654253 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:33.654226 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:32:33.654391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:33.654280 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 06:32:33.713549 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:32:33.713507 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:32:34.103385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:34.103336 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f"] Oct 30 06:32:34.661258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:34.660822 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:32:34.661258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:34.660893 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 06:32:34.747364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:34.744044 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:32:35.652682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:35.652638 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:32:35.652890 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:35.652728 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 06:32:36.652464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:36.652429 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:32:36.653040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:36.652496 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 06:32:37.652643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:37.652608 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 06:32:37.653075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:37.652678 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 06:32:38.671210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:38.671175 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:32:38.675925 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:38.673276 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:32:49.129101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.129029 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" podUID="648f6993-0732-4d7f-96ec-dc9283e60c28" containerName="main" containerID="cri-o://14f950d8b501f08a1d0404dbf9b0ca5069a3164ddbc49cc53a8e234cb9ff5792" gracePeriod=45 Oct 30 06:32:49.304516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.304471 4431 generic.go:356] "Generic (PLEG): container finished" podID="648f6993-0732-4d7f-96ec-dc9283e60c28" containerID="14f950d8b501f08a1d0404dbf9b0ca5069a3164ddbc49cc53a8e234cb9ff5792" exitCode=0 Oct 30 06:32:49.304718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.304632 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" event={"ID":"648f6993-0732-4d7f-96ec-dc9283e60c28","Type":"ContainerDied","Data":"14f950d8b501f08a1d0404dbf9b0ca5069a3164ddbc49cc53a8e234cb9ff5792"} Oct 30 06:32:49.444600 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.444573 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:32:49.475810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.475771 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt"] Oct 30 06:32:49.476513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.476492 4431 cpu_manager.go:414] "containerMap: removing container" podUID="648f6993-0732-4d7f-96ec-dc9283e60c28" containerName="main" Oct 30 06:32:49.476513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.476513 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648f6993-0732-4d7f-96ec-dc9283e60c28" containerName="main" Oct 30 06:32:49.476684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.476663 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="648f6993-0732-4d7f-96ec-dc9283e60c28" containerName="main" Oct 30 06:32:49.479887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.479865 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:32:49.498548 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.498503 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-69f3fa\"" Oct 30 06:32:49.502843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.502820 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt"] Oct 30 06:32:49.502983 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.502943 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-69f3fa-dockercfg-zpshv\"" Oct 30 06:32:49.535507 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.535470 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl"] Oct 30 06:32:49.539654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.539632 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:32:49.556561 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.556534 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl"] Oct 30 06:32:49.581178 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.581144 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-tmp-dir\") pod \"648f6993-0732-4d7f-96ec-dc9283e60c28\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " Oct 30 06:32:49.581403 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.581250 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-model-cache\") pod \"648f6993-0732-4d7f-96ec-dc9283e60c28\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " Oct 30 06:32:49.581403 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.581290 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-home\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-home\") pod \"648f6993-0732-4d7f-96ec-dc9283e60c28\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " Oct 30 06:32:49.581403 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.581343 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648f6993-0732-4d7f-96ec-dc9283e60c28-tls-certs\" (UniqueName: \"kubernetes.io/secret/648f6993-0732-4d7f-96ec-dc9283e60c28-tls-certs\") pod \"648f6993-0732-4d7f-96ec-dc9283e60c28\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " Oct 30 06:32:49.581527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.581454 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-dshm\") pod \"648f6993-0732-4d7f-96ec-dc9283e60c28\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " Oct 30 06:32:49.581572 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.581527 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648f6993-0732-4d7f-96ec-dc9283e60c28-kube-api-access-bxtf8\" (UniqueName: \"kubernetes.io/projected/648f6993-0732-4d7f-96ec-dc9283e60c28-kube-api-access-bxtf8\") pod \"648f6993-0732-4d7f-96ec-dc9283e60c28\" (UID: \"648f6993-0732-4d7f-96ec-dc9283e60c28\") " Oct 30 06:32:49.582023 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.581990 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-model-cache" pod "648f6993-0732-4d7f-96ec-dc9283e60c28" (UID: "648f6993-0732-4d7f-96ec-dc9283e60c28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:49.582183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.582027 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b9d11a-e96d-4733-b920-2ecf04bee128-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt\" (UID: \"f5b9d11a-e96d-4733-b920-2ecf04bee128\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:32:49.582434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.582410 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xncj8\" (UniqueName: \"kubernetes.io/projected/f5b9d11a-e96d-4733-b920-2ecf04bee128-kube-api-access-xncj8\") pod \"payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt\" (UID: \"f5b9d11a-e96d-4733-b920-2ecf04bee128\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:32:49.582616 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.582445 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-home" pod "648f6993-0732-4d7f-96ec-dc9283e60c28" (UID: "648f6993-0732-4d7f-96ec-dc9283e60c28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:49.582755 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.582531 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-tmp-dir" pod "648f6993-0732-4d7f-96ec-dc9283e60c28" (UID: "648f6993-0732-4d7f-96ec-dc9283e60c28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:49.582886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.582815 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.584229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.584200 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648f6993-0732-4d7f-96ec-dc9283e60c28-tls-certs" pod "648f6993-0732-4d7f-96ec-dc9283e60c28" (UID: "648f6993-0732-4d7f-96ec-dc9283e60c28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:49.584372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.584322 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-dshm" pod "648f6993-0732-4d7f-96ec-dc9283e60c28" (UID: "648f6993-0732-4d7f-96ec-dc9283e60c28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:49.584966 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.584938 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648f6993-0732-4d7f-96ec-dc9283e60c28-kube-api-access-bxtf8" pod "648f6993-0732-4d7f-96ec-dc9283e60c28" (UID: "648f6993-0732-4d7f-96ec-dc9283e60c28"). InnerVolumeSpecName "kube-api-access-bxtf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:49.710326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.697993 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wvzr\" (UniqueName: \"kubernetes.io/projected/f8efc049-5c75-4627-b33c-b0407d1a2067-kube-api-access-2wvzr\") pod \"payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl\" (UID: \"f8efc049-5c75-4627-b33c-b0407d1a2067\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:32:49.710326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.698115 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b9d11a-e96d-4733-b920-2ecf04bee128-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt\" (UID: \"f5b9d11a-e96d-4733-b920-2ecf04bee128\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:32:49.710326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.698165 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xncj8\" (UniqueName: \"kubernetes.io/projected/f5b9d11a-e96d-4733-b920-2ecf04bee128-kube-api-access-xncj8\") pod \"payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt\" (UID: \"f5b9d11a-e96d-4733-b920-2ecf04bee128\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:32:49.710326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.698213 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8efc049-5c75-4627-b33c-b0407d1a2067-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl\" (UID: \"f8efc049-5c75-4627-b33c-b0407d1a2067\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:32:49.710326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.698347 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.710326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.698366 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxtf8\" (UniqueName: \"kubernetes.io/projected/648f6993-0732-4d7f-96ec-dc9283e60c28-kube-api-access-bxtf8\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.710326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.698392 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.710326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.698408 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648f6993-0732-4d7f-96ec-dc9283e60c28-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.710326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.698420 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648f6993-0732-4d7f-96ec-dc9283e60c28-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.723724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.723636 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b9d11a-e96d-4733-b920-2ecf04bee128-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt\" (UID: \"f5b9d11a-e96d-4733-b920-2ecf04bee128\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:32:49.741549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.741459 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xncj8\" (UniqueName: \"kubernetes.io/projected/f5b9d11a-e96d-4733-b920-2ecf04bee128-kube-api-access-xncj8\") pod \"payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt\" (UID: \"f5b9d11a-e96d-4733-b920-2ecf04bee128\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:32:49.795855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.795813 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:32:49.799671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.799629 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8efc049-5c75-4627-b33c-b0407d1a2067-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl\" (UID: \"f8efc049-5c75-4627-b33c-b0407d1a2067\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:32:49.799857 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.799837 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wvzr\" (UniqueName: \"kubernetes.io/projected/f8efc049-5c75-4627-b33c-b0407d1a2067-kube-api-access-2wvzr\") pod \"payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl\" (UID: \"f8efc049-5c75-4627-b33c-b0407d1a2067\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:32:49.800648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.800618 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8efc049-5c75-4627-b33c-b0407d1a2067-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl\" (UID: \"f8efc049-5c75-4627-b33c-b0407d1a2067\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:32:49.823032 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.822996 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wvzr\" (UniqueName: \"kubernetes.io/projected/f8efc049-5c75-4627-b33c-b0407d1a2067-kube-api-access-2wvzr\") pod \"payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl\" (UID: \"f8efc049-5c75-4627-b33c-b0407d1a2067\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:32:49.860811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:49.859290 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:32:50.111510 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:32:50.111481 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b9d11a_e96d_4733_b920_2ecf04bee128.slice/crio-35e30903bf0ec0db5a4400055685331f6d557f3965e7aaf3d280af5dea5fc2c4 WatchSource:0}: Error finding container 35e30903bf0ec0db5a4400055685331f6d557f3965e7aaf3d280af5dea5fc2c4: Status 404 returned error can't find the container with id 35e30903bf0ec0db5a4400055685331f6d557f3965e7aaf3d280af5dea5fc2c4 Oct 30 06:32:50.111688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.111665 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt"] Oct 30 06:32:50.196795 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:32:50.196761 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8efc049_5c75_4627_b33c_b0407d1a2067.slice/crio-cfb3bfbe9120642e07e199d379ed26d099c2cf21041027ad30a2022a7334e6ac WatchSource:0}: Error finding container cfb3bfbe9120642e07e199d379ed26d099c2cf21041027ad30a2022a7334e6ac: Status 404 returned error can't find the container with id cfb3bfbe9120642e07e199d379ed26d099c2cf21041027ad30a2022a7334e6ac Oct 30 06:32:50.198199 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.198176 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl"] Oct 30 06:32:50.313086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.313049 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" event={"ID":"f5b9d11a-e96d-4733-b920-2ecf04bee128","Type":"ContainerStarted","Data":"48bbca1a8d749e10307bbbc3858d2c478da536b89c4584326d7a7674f3c103e3"} Oct 30 06:32:50.313313 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.313108 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" event={"ID":"f5b9d11a-e96d-4733-b920-2ecf04bee128","Type":"ContainerStarted","Data":"35e30903bf0ec0db5a4400055685331f6d557f3965e7aaf3d280af5dea5fc2c4"} Oct 30 06:32:50.313313 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.313136 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:32:50.314779 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.314759 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" Oct 30 06:32:50.314904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.314808 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f" event={"ID":"648f6993-0732-4d7f-96ec-dc9283e60c28","Type":"ContainerDied","Data":"b43ea2d141c22bbfa37538cd10cedda16dd0f61ce10a5e53c124b375c1b8a8d5"} Oct 30 06:32:50.314904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.314859 4431 scope.go:122] "RemoveContainer" containerID="14f950d8b501f08a1d0404dbf9b0ca5069a3164ddbc49cc53a8e234cb9ff5792" Oct 30 06:32:50.316534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.316511 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" event={"ID":"f8efc049-5c75-4627-b33c-b0407d1a2067","Type":"ContainerStarted","Data":"e07b7acb537e23ce9d5a725f52d573448803ecceac78666421af141584755bd3"} Oct 30 06:32:50.316660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.316566 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:32:50.316660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.316592 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" event={"ID":"f8efc049-5c75-4627-b33c-b0407d1a2067","Type":"ContainerStarted","Data":"cfb3bfbe9120642e07e199d379ed26d099c2cf21041027ad30a2022a7334e6ac"} Oct 30 06:32:50.354617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.354565 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" podStartSLOduration=1.354548025 podStartE2EDuration="1.354548025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:50.35307877 +0000 UTC m=+2379.159600519" watchObservedRunningTime="2026-10-30 06:32:50.354548025 +0000 UTC m=+2379.161069773" Oct 30 06:32:50.397201 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.397094 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" podStartSLOduration=1.397077334 podStartE2EDuration="1.397077334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:50.395139417 +0000 UTC m=+2379.201661166" watchObservedRunningTime="2026-10-30 06:32:50.397077334 +0000 UTC m=+2379.203599082" Oct 30 06:32:50.433985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.433953 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f"] Oct 30 06:32:50.436897 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:50.436872 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-30fbb6/auth-test-model-30fbb6-kserve-79665d489-hjh6f"] Oct 30 06:32:51.339774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:32:51.339742 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648f6993-0732-4d7f-96ec-dc9283e60c28" path="/var/lib/kubelet/pods/648f6993-0732-4d7f-96ec-dc9283e60c28/volumes" Oct 30 06:33:01.327316 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:01.327205 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:33:01.329660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:01.329636 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:33:03.703386 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:33:03.703349 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:33:13.945325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:13.937715 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns"] Oct 30 06:33:29.051847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.051772 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" podUID="ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" containerName="main" containerID="cri-o://23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40" gracePeriod=45 Oct 30 06:33:29.329915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.329886 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:33:29.416347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.416297 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tls-certs\") pod \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " Oct 30 06:33:29.416559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.416374 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-home\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-home\") pod \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " Oct 30 06:33:29.416559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.416521 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tmp-dir\") pod \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " Oct 30 06:33:29.416559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.416547 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-kube-api-access-h7k52\" (UniqueName: \"kubernetes.io/projected/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-kube-api-access-h7k52\") pod \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " Oct 30 06:33:29.416718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.416603 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-model-cache\") pod \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " Oct 30 06:33:29.416718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.416621 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-home" pod "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" (UID: "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:29.416798 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.416718 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-dshm\") pod \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\" (UID: \"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21\") " Oct 30 06:33:29.416798 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.416778 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tmp-dir" pod "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" (UID: "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:29.416951 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.416923 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-model-cache" pod "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" (UID: "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:29.417270 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.417247 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:33:29.417373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.417272 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:33:29.417373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.417283 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:33:29.418714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.418693 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-dshm" pod "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" (UID: "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:29.418840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.418798 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-kube-api-access-h7k52" pod "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" (UID: "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21"). InnerVolumeSpecName "kube-api-access-h7k52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:29.418840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.418810 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tls-certs" pod "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" (UID: "ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:33:29.518756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.518717 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7k52\" (UniqueName: \"kubernetes.io/projected/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-kube-api-access-h7k52\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:33:29.518756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.518747 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:33:29.518756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.518756 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:33:29.581734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.581637 4431 generic.go:356] "Generic (PLEG): container finished" podID="ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" containerID="23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40" exitCode=0 Oct 30 06:33:29.581734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.581695 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" Oct 30 06:33:29.581956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.581725 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" event={"ID":"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21","Type":"ContainerDied","Data":"23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40"} Oct 30 06:33:29.581956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.581770 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns" event={"ID":"ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21","Type":"ContainerDied","Data":"7403e406114d85647df30f3decaccaebe446baaec88b72e0401ac2d2458a07b0"} Oct 30 06:33:29.581956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.581789 4431 scope.go:122] "RemoveContainer" containerID="23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40" Oct 30 06:33:29.593220 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.593201 4431 scope.go:122] "RemoveContainer" containerID="23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40" Oct 30 06:33:29.593547 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:33:29.593527 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40\": container with ID starting with 23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40 not found: ID does not exist" containerID="23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40" Oct 30 06:33:29.593627 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.593561 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40"} err="failed to get container status \"23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40\": rpc error: code = NotFound desc = could not find container \"23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40\": container with ID starting with 23793eb559ac0aa2d36dcdba8bb99a31b0e05b56c157260e20c0e7a158f4bb40 not found: ID does not exist" Oct 30 06:33:29.633006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.632963 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns"] Oct 30 06:33:29.637475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:29.637440 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4f3973/auth-test-model-4f3973-kserve-9ccbbb56f-fm8ns"] Oct 30 06:33:31.350471 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:31.350423 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" path="/var/lib/kubelet/pods/ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21/volumes" Oct 30 06:33:33.708042 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:33:33.707995 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:33:36.363236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.363202 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7"] Oct 30 06:33:36.363859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.363843 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" containerName="main" Oct 30 06:33:36.363903 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.363860 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" containerName="main" Oct 30 06:33:36.363969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.363960 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee10f863-2e1b-4c4a-8d5d-0ab8ce4c9b21" containerName="main" Oct 30 06:33:36.368699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.368678 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:36.388541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.388510 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7"] Oct 30 06:33:36.396271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.396244 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a88d7d\"" Oct 30 06:33:36.428033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.427998 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8"] Oct 30 06:33:36.433243 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.433219 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:36.447947 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.447917 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a88d7d-dockercfg-9ggl2\"" Oct 30 06:33:36.448496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.448476 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8"] Oct 30 06:33:36.456795 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.456766 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6208c4e3-03cf-4149-bdad-57b37eb2b759-plugins-config-volume\") pod \"payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7\" (UID: \"6208c4e3-03cf-4149-bdad-57b37eb2b759\") " pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:36.457072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.457031 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxwk\" (UniqueName: \"kubernetes.io/projected/6208c4e3-03cf-4149-bdad-57b37eb2b759-kube-api-access-2kxwk\") pod \"payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7\" (UID: \"6208c4e3-03cf-4149-bdad-57b37eb2b759\") " pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:36.568286 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.568239 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzkl\" (UniqueName: \"kubernetes.io/projected/3f801576-e5e0-4065-bfc7-bb74f7c3e839-kube-api-access-lpzkl\") pod \"payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8\" (UID: \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:36.568779 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.568749 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxwk\" (UniqueName: \"kubernetes.io/projected/6208c4e3-03cf-4149-bdad-57b37eb2b759-kube-api-access-2kxwk\") pod \"payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7\" (UID: \"6208c4e3-03cf-4149-bdad-57b37eb2b759\") " pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:36.569393 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.569371 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6208c4e3-03cf-4149-bdad-57b37eb2b759-plugins-config-volume\") pod \"payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7\" (UID: \"6208c4e3-03cf-4149-bdad-57b37eb2b759\") " pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:36.570451 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.570432 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f801576-e5e0-4065-bfc7-bb74f7c3e839-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8\" (UID: \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:36.570710 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.570348 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6208c4e3-03cf-4149-bdad-57b37eb2b759-plugins-config-volume\") pod \"payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7\" (UID: \"6208c4e3-03cf-4149-bdad-57b37eb2b759\") " pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:36.603720 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.603635 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxwk\" (UniqueName: \"kubernetes.io/projected/6208c4e3-03cf-4149-bdad-57b37eb2b759-kube-api-access-2kxwk\") pod \"payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7\" (UID: \"6208c4e3-03cf-4149-bdad-57b37eb2b759\") " pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:36.672115 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.672024 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f801576-e5e0-4065-bfc7-bb74f7c3e839-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8\" (UID: \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:36.672115 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.672098 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzkl\" (UniqueName: \"kubernetes.io/projected/3f801576-e5e0-4065-bfc7-bb74f7c3e839-kube-api-access-lpzkl\") pod \"payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8\" (UID: \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:36.672804 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.672783 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f801576-e5e0-4065-bfc7-bb74f7c3e839-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8\" (UID: \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:36.682658 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.682628 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:36.695795 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.695765 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzkl\" (UniqueName: \"kubernetes.io/projected/3f801576-e5e0-4065-bfc7-bb74f7c3e839-kube-api-access-lpzkl\") pod \"payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8\" (UID: \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:36.747781 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.747508 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:36.991578 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:33:36.991547 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6208c4e3_03cf_4149_bdad_57b37eb2b759.slice/crio-8e5d90cf8355d71a2594980c610fb0684ea4eca1db8b96e08a21d17c45ea82b1 WatchSource:0}: Error finding container 8e5d90cf8355d71a2594980c610fb0684ea4eca1db8b96e08a21d17c45ea82b1: Status 404 returned error can't find the container with id 8e5d90cf8355d71a2594980c610fb0684ea4eca1db8b96e08a21d17c45ea82b1 Oct 30 06:33:36.992207 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:36.992183 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7"] Oct 30 06:33:37.080411 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:33:37.080383 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f801576_e5e0_4065_bfc7_bb74f7c3e839.slice/crio-4a163df5abc5971968a3d8a1f6e2028d1f38e13e1b776ad6ed428213a425629e WatchSource:0}: Error finding container 4a163df5abc5971968a3d8a1f6e2028d1f38e13e1b776ad6ed428213a425629e: Status 404 returned error can't find the container with id 4a163df5abc5971968a3d8a1f6e2028d1f38e13e1b776ad6ed428213a425629e Oct 30 06:33:37.080766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:37.080745 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8"] Oct 30 06:33:37.641557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:37.641525 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" event={"ID":"3f801576-e5e0-4065-bfc7-bb74f7c3e839","Type":"ContainerStarted","Data":"8c6b0aa5845dcc8e97566614f913e23cc5a5fec7ce2a4f89c4a03a48b60f2869"} Oct 30 06:33:37.642029 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:37.641583 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" event={"ID":"3f801576-e5e0-4065-bfc7-bb74f7c3e839","Type":"ContainerStarted","Data":"4a163df5abc5971968a3d8a1f6e2028d1f38e13e1b776ad6ed428213a425629e"} Oct 30 06:33:37.642029 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:37.641607 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:37.643781 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:37.643727 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" event={"ID":"6208c4e3-03cf-4149-bdad-57b37eb2b759","Type":"ContainerStarted","Data":"972c2b089a145348a713a5824e37cbf4ea3a46301fbbc1461ba506d17cef702c"} Oct 30 06:33:37.643980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:37.643782 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" event={"ID":"6208c4e3-03cf-4149-bdad-57b37eb2b759","Type":"ContainerStarted","Data":"8e5d90cf8355d71a2594980c610fb0684ea4eca1db8b96e08a21d17c45ea82b1"} Oct 30 06:33:37.643980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:37.643831 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:37.686457 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:37.686410 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" podStartSLOduration=1.6863975519999999 podStartE2EDuration="1.686397552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:37.683979609 +0000 UTC m=+2426.490501357" watchObservedRunningTime="2026-10-30 06:33:37.686397552 +0000 UTC m=+2426.492919300" Oct 30 06:33:37.730022 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:37.729978 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" podStartSLOduration=1.72996569 podStartE2EDuration="1.72996569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:37.727878166 +0000 UTC m=+2426.534399916" watchObservedRunningTime="2026-10-30 06:33:37.72996569 +0000 UTC m=+2426.536487482" Oct 30 06:33:40.087404 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.087372 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8"] Oct 30 06:33:40.091544 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.091523 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.108521 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.108503 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a0083e-openshift-default-dockercfg-zxj88\"" Oct 30 06:33:40.122648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.122618 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8"] Oct 30 06:33:40.214870 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.214836 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.215010 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.214895 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4m4\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-kube-api-access-4w4m4\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.215055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.215039 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.215091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.215074 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.215144 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.215090 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.215200 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.215186 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.215243 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.215233 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.215403 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.215385 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.215469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.215431 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4613cd6-89a5-49aa-aca4-566a8605cce7-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.316939 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.316904 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.316978 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317003 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4613cd6-89a5-49aa-aca4-566a8605cce7-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317032 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317060 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4m4\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-kube-api-access-4w4m4\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317135 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317157 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317178 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317216 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317540 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317631 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317690 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317704 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.317935 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.317912 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4613cd6-89a5-49aa-aca4-566a8605cce7-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.319680 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.319662 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.319896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.319879 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.340040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.339981 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.340141 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.340075 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4m4\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-kube-api-access-4w4m4\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.405776 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.405745 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:40.721021 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:33:40.720994 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4613cd6_89a5_49aa_aca4_566a8605cce7.slice/crio-621fa711c9137f51fde216a455bf5a4cc68ba71c3f8c5c327891485fdd1d1676 WatchSource:0}: Error finding container 621fa711c9137f51fde216a455bf5a4cc68ba71c3f8c5c327891485fdd1d1676: Status 404 returned error can't find the container with id 621fa711c9137f51fde216a455bf5a4cc68ba71c3f8c5c327891485fdd1d1676 Oct 30 06:33:40.721772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.721743 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8"] Oct 30 06:33:40.723831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.723801 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:33:40.723889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.723866 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:33:40.723924 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:40.723895 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:33:41.684065 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:41.684026 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" event={"ID":"c4613cd6-89a5-49aa-aca4-566a8605cce7","Type":"ContainerStarted","Data":"e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e"} Oct 30 06:33:41.684488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:41.684087 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" event={"ID":"c4613cd6-89a5-49aa-aca4-566a8605cce7","Type":"ContainerStarted","Data":"621fa711c9137f51fde216a455bf5a4cc68ba71c3f8c5c327891485fdd1d1676"} Oct 30 06:33:41.743519 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:41.743470 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" podStartSLOduration=1.743457225 podStartE2EDuration="1.743457225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:41.741998978 +0000 UTC m=+2430.548520750" watchObservedRunningTime="2026-10-30 06:33:41.743457225 +0000 UTC m=+2430.549978973" Oct 30 06:33:42.405974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:42.405934 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:42.407381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:42.407356 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:33:42.407504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:42.407405 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 06:33:43.406996 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:43.406964 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:33:43.407507 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:43.407021 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 06:33:44.406383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:44.406349 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:33:44.406559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:44.406409 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 06:33:45.407061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:45.407026 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:33:45.407582 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:45.407103 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 06:33:46.406016 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:46.405979 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:33:46.406205 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:46.406035 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 06:33:47.469761 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:47.469701 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:47.470755 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:47.470574 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:47.473876 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:47.473855 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:33:48.653892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:48.653864 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:33:48.654360 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:48.653926 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:33:55.153332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.153288 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl"] Oct 30 06:33:55.157601 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.157584 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.190894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.190868 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl"] Oct 30 06:33:55.267933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.267898 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207c013b-e701-44ae-8363-df664833ec83-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.268093 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.267955 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.268093 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.268058 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.268093 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.268086 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.268217 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.268180 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.268265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.268241 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.268346 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.268290 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-kube-api-access-mrh44\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.268443 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.268426 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.268498 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.268470 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207c013b-e701-44ae-8363-df664833ec83-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.369459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.369427 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.369459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.369466 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207c013b-e701-44ae-8363-df664833ec83-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.369648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.369527 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207c013b-e701-44ae-8363-df664833ec83-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.369648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.369550 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.369648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.369603 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.369648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.369623 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.369822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.369682 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.369822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.369731 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.369822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.369796 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-kube-api-access-mrh44\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.370083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.370051 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.370229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.370205 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.370352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.370330 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.370420 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.370356 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.370420 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.370390 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207c013b-e701-44ae-8363-df664833ec83-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.371865 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.371843 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.372248 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.372224 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207c013b-e701-44ae-8363-df664833ec83-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.392635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.392611 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.392782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.392760 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-kube-api-access-mrh44\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.473000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.472973 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:55.596518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.596487 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx"] Oct 30 06:33:55.601876 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.601856 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.618664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.618636 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ad1661-openshift-default-dockercfg-2bbrc\"" Oct 30 06:33:55.622652 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.622625 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx"] Oct 30 06:33:55.674796 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.674764 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.675003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.674802 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6szt\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-kube-api-access-p6szt\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.675003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.674957 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e912eea-45ff-47e0-b15b-165a912ce731-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.675003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.674985 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e912eea-45ff-47e0-b15b-165a912ce731-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.675159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.675031 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.675159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.675078 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.675268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.675220 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.675358 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.675333 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.675434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.675415 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.777232 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.777153 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e912eea-45ff-47e0-b15b-165a912ce731-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.777232 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.777188 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e912eea-45ff-47e0-b15b-165a912ce731-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.777232 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.777219 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.777527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.777250 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.777527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.777297 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.777527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.777460 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.777743 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.777688 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.777870 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.777810 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.777938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.777875 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.778032 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.778004 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.778182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.778058 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e912eea-45ff-47e0-b15b-165a912ce731-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.778182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.778132 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.778182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.778160 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6szt\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-kube-api-access-p6szt\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.778362 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.778318 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.780165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.780142 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.780511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.780490 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e912eea-45ff-47e0-b15b-165a912ce731-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.790971 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:33:55.790947 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207c013b_e701_44ae_8363_df664833ec83.slice/crio-7b9b71db46381d89c5c717c7a4da48ac9213b9c6a90e6c53c262a21b331b6a1d WatchSource:0}: Error finding container 7b9b71db46381d89c5c717c7a4da48ac9213b9c6a90e6c53c262a21b331b6a1d: Status 404 returned error can't find the container with id 7b9b71db46381d89c5c717c7a4da48ac9213b9c6a90e6c53c262a21b331b6a1d Oct 30 06:33:55.792004 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.791961 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl"] Oct 30 06:33:55.794592 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.794552 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:33:55.794744 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.794639 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:33:55.794744 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.794683 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:33:55.805061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.805037 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.805491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.805472 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6szt\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-kube-api-access-p6szt\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:55.917252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:55.917222 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:56.336067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.336023 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx"] Oct 30 06:33:56.348459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.347749 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:33:56.348459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.347850 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:33:56.348459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.347905 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:33:56.795722 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.795687 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" event={"ID":"207c013b-e701-44ae-8363-df664833ec83","Type":"ContainerStarted","Data":"fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242"} Oct 30 06:33:56.795894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.795740 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" event={"ID":"207c013b-e701-44ae-8363-df664833ec83","Type":"ContainerStarted","Data":"7b9b71db46381d89c5c717c7a4da48ac9213b9c6a90e6c53c262a21b331b6a1d"} Oct 30 06:33:56.797500 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.797464 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" event={"ID":"8e912eea-45ff-47e0-b15b-165a912ce731","Type":"ContainerStarted","Data":"f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07"} Oct 30 06:33:56.797630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.797530 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" event={"ID":"8e912eea-45ff-47e0-b15b-165a912ce731","Type":"ContainerStarted","Data":"7c1cb126b525be1783e3f7be246538c9ad04e31f2c1cbd835abb1925b12f7533"} Oct 30 06:33:56.842254 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.842203 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" podStartSLOduration=1.842188046 podStartE2EDuration="1.842188046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:56.840206609 +0000 UTC m=+2445.646728358" watchObservedRunningTime="2026-10-30 06:33:56.842188046 +0000 UTC m=+2445.648709793" Oct 30 06:33:56.893422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.893363 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" podStartSLOduration=1.893292124 podStartE2EDuration="1.893292124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:56.890206374 +0000 UTC m=+2445.696728122" watchObservedRunningTime="2026-10-30 06:33:56.893292124 +0000 UTC m=+2445.699813872" Oct 30 06:33:56.917762 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:56.917735 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:33:57.495327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:57.478925 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:33:57.495327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:57.489868 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:33:57.495327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:57.489923 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" podUID="207c013b-e701-44ae-8363-df664833ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 06:33:57.921105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:57.921028 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 06:33:57.921105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:57.921076 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 06:33:58.479321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:58.478703 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:33:58.479321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:58.478779 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" podUID="207c013b-e701-44ae-8363-df664833ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 06:33:58.918603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:58.918515 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 06:33:58.918603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:58.918580 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 06:33:59.475799 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:59.475759 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:33:59.476100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:59.476059 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" podUID="207c013b-e701-44ae-8363-df664833ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 06:33:59.917957 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:59.917872 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 06:33:59.917957 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:33:59.917930 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 06:34:00.478327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:00.475711 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:34:00.478327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:00.475775 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" podUID="207c013b-e701-44ae-8363-df664833ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 06:34:00.918481 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:00.918405 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 06:34:00.918669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:00.918649 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 06:34:01.473799 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:01.473750 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 06:34:01.474089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:01.474060 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" podUID="207c013b-e701-44ae-8363-df664833ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 06:34:01.917909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:01.917834 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 06:34:01.918377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:01.918345 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 06:34:02.727290 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:02.727254 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:34:02.728529 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:02.728208 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:34:02.729256 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:02.729150 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:34:02.918444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:02.918411 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 06:34:02.918896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:02.918469 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 06:34:03.702916 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:34:03.702884 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:34:03.924157 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:03.924125 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:34:03.924791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:03.924772 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:34:03.925474 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:03.925454 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:34:04.153495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.153411 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz"] Oct 30 06:34:04.158738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.158712 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.188928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.188883 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6d1778-openshift-default-dockercfg-5rlck\"" Oct 30 06:34:04.190593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.190411 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz"] Oct 30 06:34:04.242079 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.242036 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-credential-socket\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.242079 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.242086 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-certs\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.242334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.242154 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-data\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.242334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.242281 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-token\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.242458 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.242405 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-podinfo\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.242511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.242477 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-socket\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.242549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.242508 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwnr\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-kube-api-access-5lwnr\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.242585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.242552 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istiod-ca-cert\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.242630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.242607 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-envoy\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344278 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-socket\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344323 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwnr\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-kube-api-access-5lwnr\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344350 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istiod-ca-cert\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344411 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-envoy\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344541 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-credential-socket\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344578 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-certs\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344625 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-data\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344662 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-socket\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344702 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-token\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.344984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344839 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-podinfo\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.345043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.344988 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-credential-socket\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.345043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.345003 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-certs\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.345117 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.345098 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istiod-ca-cert\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.345283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.345260 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-data\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.346751 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.346729 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-envoy\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.347107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.347090 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-podinfo\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.368101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.368073 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-token\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.368195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.368144 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwnr\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-kube-api-access-5lwnr\") pod \"e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.477127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.477099 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:04.952060 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:34:04.952033 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253e2111_a9ac_4ddf_a3f8_6e86965c99aa.slice/crio-377577f7ea6ae3cbb546728b68254202d09459d95f216efd497d7fb5b0d4b1b0 WatchSource:0}: Error finding container 377577f7ea6ae3cbb546728b68254202d09459d95f216efd497d7fb5b0d4b1b0: Status 404 returned error can't find the container with id 377577f7ea6ae3cbb546728b68254202d09459d95f216efd497d7fb5b0d4b1b0 Oct 30 06:34:04.952625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.952560 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz"] Oct 30 06:34:04.955274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.955236 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:04.955396 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.955333 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:04.955396 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:04.955377 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:05.886955 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:05.886910 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" event={"ID":"253e2111-a9ac-4ddf-a3f8-6e86965c99aa","Type":"ContainerStarted","Data":"abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4"} Oct 30 06:34:05.887138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:05.886972 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" event={"ID":"253e2111-a9ac-4ddf-a3f8-6e86965c99aa","Type":"ContainerStarted","Data":"377577f7ea6ae3cbb546728b68254202d09459d95f216efd497d7fb5b0d4b1b0"} Oct 30 06:34:05.950876 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:05.950827 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podStartSLOduration=1.950812596 podStartE2EDuration="1.950812596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:05.938760447 +0000 UTC m=+2454.745282233" watchObservedRunningTime="2026-10-30 06:34:05.950812596 +0000 UTC m=+2454.757334344" Oct 30 06:34:06.477712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:06.477681 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:06.484408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:06.484375 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:06.893182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:06.893105 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:06.894398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:06.894377 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:34:11.190504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.190464 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr"] Oct 30 06:34:11.195816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.195794 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:11.210212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.210180 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr"] Oct 30 06:34:11.253643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.253609 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6d1778\"" Oct 30 06:34:11.257682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.256223 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6d1778-dockercfg-bhk94\"" Oct 30 06:34:11.321326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.317995 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvzv\" (UniqueName: \"kubernetes.io/projected/03aa02fa-ef29-4efb-ae52-5ed210c4a404-kube-api-access-8zvzv\") pod \"payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr\" (UID: \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:11.321326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.318096 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa02fa-ef29-4efb-ae52-5ed210c4a404-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr\" (UID: \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:11.440236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.440195 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvzv\" (UniqueName: \"kubernetes.io/projected/03aa02fa-ef29-4efb-ae52-5ed210c4a404-kube-api-access-8zvzv\") pod \"payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr\" (UID: \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:11.440587 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.440530 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa02fa-ef29-4efb-ae52-5ed210c4a404-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr\" (UID: \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:11.464387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.455144 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6d1778\"" Oct 30 06:34:11.464611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.456259 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa02fa-ef29-4efb-ae52-5ed210c4a404-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr\" (UID: \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:11.469673 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.469432 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvzv\" (UniqueName: \"kubernetes.io/projected/03aa02fa-ef29-4efb-ae52-5ed210c4a404-kube-api-access-8zvzv\") pod \"payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr\" (UID: \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:11.533905 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.533864 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6d1778-dockercfg-bhk94\"" Oct 30 06:34:11.540974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.540945 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:11.844090 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:34:11.844058 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03aa02fa_ef29_4efb_ae52_5ed210c4a404.slice/crio-1d02da6a3898838ee079f24f3b89ba137753838b346bb0e5818937736ba6f197 WatchSource:0}: Error finding container 1d02da6a3898838ee079f24f3b89ba137753838b346bb0e5818937736ba6f197: Status 404 returned error can't find the container with id 1d02da6a3898838ee079f24f3b89ba137753838b346bb0e5818937736ba6f197 Oct 30 06:34:11.844288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.844268 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr"] Oct 30 06:34:11.955376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.955347 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" event={"ID":"03aa02fa-ef29-4efb-ae52-5ed210c4a404","Type":"ContainerStarted","Data":"cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80"} Oct 30 06:34:11.955522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.955400 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" event={"ID":"03aa02fa-ef29-4efb-ae52-5ed210c4a404","Type":"ContainerStarted","Data":"1d02da6a3898838ee079f24f3b89ba137753838b346bb0e5818937736ba6f197"} Oct 30 06:34:11.955522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.955427 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:11.995292 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:11.995237 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" podStartSLOduration=0.995220217 podStartE2EDuration="995.220217ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:11.994555161 +0000 UTC m=+2460.801076913" watchObservedRunningTime="2026-10-30 06:34:11.995220217 +0000 UTC m=+2460.801741966" Oct 30 06:34:22.975886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:22.975856 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:34:25.265886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.265805 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg"] Oct 30 06:34:25.270421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.270388 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.301091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.301058 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg"] Oct 30 06:34:25.371956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.371928 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7fq\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-kube-api-access-kq7fq\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.372122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.372013 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.372122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.372057 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.372122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.372077 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728404ea-ba21-420d-bc69-170c41334445-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.372227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.372188 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.372263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.372234 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.372335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.372293 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.372483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.372465 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728404ea-ba21-420d-bc69-170c41334445-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.372522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.372513 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.473872 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.473838 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474065 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.473992 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474065 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474026 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728404ea-ba21-420d-bc69-170c41334445-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474076 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474239 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474180 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474239 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474223 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474396 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474264 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474396 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474353 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474423 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728404ea-ba21-420d-bc69-170c41334445-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474491 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474636 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474591 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7fq\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-kube-api-access-kq7fq\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474638 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.474875 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.474850 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.475024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.475003 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728404ea-ba21-420d-bc69-170c41334445-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.476534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.476511 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.476659 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.476641 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728404ea-ba21-420d-bc69-170c41334445-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.497251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.497221 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7fq\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-kube-api-access-kq7fq\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.497251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.497238 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.600916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.600885 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:25.949657 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.949622 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg"] Oct 30 06:34:25.958518 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:34:25.958481 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728404ea_ba21_420d_bc69_170c41334445.slice/crio-e714470a65cef45113e53ddef1ba30cf6a6566e41e32afa163f64a3fe8d3ab2f WatchSource:0}: Error finding container e714470a65cef45113e53ddef1ba30cf6a6566e41e32afa163f64a3fe8d3ab2f: Status 404 returned error can't find the container with id e714470a65cef45113e53ddef1ba30cf6a6566e41e32afa163f64a3fe8d3ab2f Oct 30 06:34:25.963478 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.963435 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:25.963620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.963557 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:25.963620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:25.963613 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:26.167538 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:26.167499 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" event={"ID":"728404ea-ba21-420d-bc69-170c41334445","Type":"ContainerStarted","Data":"c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4"} Oct 30 06:34:26.167713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:26.167574 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" event={"ID":"728404ea-ba21-420d-bc69-170c41334445","Type":"ContainerStarted","Data":"e714470a65cef45113e53ddef1ba30cf6a6566e41e32afa163f64a3fe8d3ab2f"} Oct 30 06:34:26.214587 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:26.214479 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" podStartSLOduration=1.214459581 podStartE2EDuration="1.214459581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:26.21169564 +0000 UTC m=+2475.018217408" watchObservedRunningTime="2026-10-30 06:34:26.214459581 +0000 UTC m=+2475.020981330" Oct 30 06:34:26.601189 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:26.601108 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:27.603028 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:27.602988 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 06:34:27.603444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:27.603046 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" podUID="728404ea-ba21-420d-bc69-170c41334445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 06:34:28.601346 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:28.601283 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 06:34:28.601520 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:28.601364 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" podUID="728404ea-ba21-420d-bc69-170c41334445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 06:34:29.602063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:29.602029 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 06:34:29.602485 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:29.602088 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" podUID="728404ea-ba21-420d-bc69-170c41334445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 06:34:30.601354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:30.601322 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 06:34:30.601528 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:30.601376 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" podUID="728404ea-ba21-420d-bc69-170c41334445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 06:34:31.601316 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:31.601271 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 06:34:31.601747 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:31.601369 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" podUID="728404ea-ba21-420d-bc69-170c41334445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 06:34:32.605744 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:32.605706 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:32.606207 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:32.606160 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:32.606825 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:32.606804 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:34:33.701946 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:34:33.701908 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:34:40.345721 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.345692 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9"] Oct 30 06:34:40.350015 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.349997 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.375618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.375589 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9"] Oct 30 06:34:40.519983 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.519941 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.520171 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.520091 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.520252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.520193 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.520347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.520295 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.520394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.520373 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.520649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.520632 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.520723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.520703 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.520766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.520748 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.520817 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.520802 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjx2m\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-kube-api-access-wjx2m\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.622896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.622808 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.622895 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.622933 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623190 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.623076 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjx2m\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-kube-api-access-wjx2m\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623243 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.623208 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623312 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.623255 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623312 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.623266 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623450 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.623425 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.623550 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.623603 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.623639 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.623975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.623951 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.624062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.624024 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.624535 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.624508 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.625965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.625939 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.626447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.626425 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.646601 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.646570 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.646712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.646599 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjx2m\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-kube-api-access-wjx2m\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.670681 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.670657 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:40.744647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.744615 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l"] Oct 30 06:34:40.752715 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.752690 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.770065 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.770033 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l"] Oct 30 06:34:40.777618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.777596 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt"] Oct 30 06:34:40.792046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.792011 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.829431 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.829403 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt"] Oct 30 06:34:40.927236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927153 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.927236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927194 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.927476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927348 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.927476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927371 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.927476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927410 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.927662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927546 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vqk\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-kube-api-access-f9vqk\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.927662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927615 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr6dk\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-kube-api-access-jr6dk\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.927773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927672 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.927773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927705 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd098f33-cfd7-4505-8abd-0100edc4feef-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.927773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927726 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.927928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927804 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f462dfd-9c1e-451c-9a45-07a936cccb60-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.927928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927831 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.927928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927846 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.928083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927940 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.928083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927971 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.928083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.927991 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:40.928083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.928039 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.928083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.928057 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:40.987931 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:34:40.987903 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdf6871_2729_4b45_9b72_4b2bcd16d1bd.slice/crio-d6a8dd8d2682370e5157d883c4aab0c1e108ab6d3c7886fda929d4c082827b2b WatchSource:0}: Error finding container d6a8dd8d2682370e5157d883c4aab0c1e108ab6d3c7886fda929d4c082827b2b: Status 404 returned error can't find the container with id d6a8dd8d2682370e5157d883c4aab0c1e108ab6d3c7886fda929d4c082827b2b Oct 30 06:34:40.989133 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.989113 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9"] Oct 30 06:34:40.990773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.990740 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:40.990872 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.990815 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:40.990872 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:40.990864 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:41.029543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029516 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vqk\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-kube-api-access-f9vqk\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.029643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029581 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr6dk\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-kube-api-access-jr6dk\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.029725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029704 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.029783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029755 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd098f33-cfd7-4505-8abd-0100edc4feef-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.029834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029780 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.029921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029840 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f462dfd-9c1e-451c-9a45-07a936cccb60-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.029921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029878 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.029921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029903 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.030090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029952 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.030090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.029982 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.030090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030008 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.030090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030048 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.030090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030075 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.030371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030172 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.030371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030210 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.030371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030335 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.030371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030366 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.030816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030391 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.030816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030419 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.030816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030665 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f462dfd-9c1e-451c-9a45-07a936cccb60-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.030816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030769 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.031019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030668 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd098f33-cfd7-4505-8abd-0100edc4feef-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.031019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030943 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.031019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.030174 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.031194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.031024 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.031252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.031195 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.031334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.031261 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.031617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.031575 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.032955 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.032934 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.033044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.033025 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.033107 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.033087 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.033647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.033626 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.053076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.053048 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.053237 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.053215 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr6dk\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-kube-api-access-jr6dk\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.053289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.053215 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vqk\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-kube-api-access-f9vqk\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.053921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.053898 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.077024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.077000 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:41.138037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.137785 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:41.285240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.285206 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" event={"ID":"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd","Type":"ContainerStarted","Data":"4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4"} Oct 30 06:34:41.285441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.285267 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" event={"ID":"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd","Type":"ContainerStarted","Data":"d6a8dd8d2682370e5157d883c4aab0c1e108ab6d3c7886fda929d4c082827b2b"} Oct 30 06:34:41.331095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.331028 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" podStartSLOduration=1.331009753 podStartE2EDuration="1.331009753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:41.329294881 +0000 UTC m=+2490.135816630" watchObservedRunningTime="2026-10-30 06:34:41.331009753 +0000 UTC m=+2490.137531504" Oct 30 06:34:41.396800 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.396768 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l"] Oct 30 06:34:41.397161 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:34:41.397131 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f462dfd_9c1e_451c_9a45_07a936cccb60.slice/crio-6e48121dfac0c41d89c2669635decd1dfa1cca6ed51a7e4f417c0d3f25c3e1ac WatchSource:0}: Error finding container 6e48121dfac0c41d89c2669635decd1dfa1cca6ed51a7e4f417c0d3f25c3e1ac: Status 404 returned error can't find the container with id 6e48121dfac0c41d89c2669635decd1dfa1cca6ed51a7e4f417c0d3f25c3e1ac Oct 30 06:34:41.400434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.400393 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:41.400557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.400481 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:41.400557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.400521 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:41.470871 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:34:41.470840 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd098f33_cfd7_4505_8abd_0100edc4feef.slice/crio-503410b054f4f54772ebf1b2e056b38f029a173cf83b9d2ce7cebbb7f7528abd WatchSource:0}: Error finding container 503410b054f4f54772ebf1b2e056b38f029a173cf83b9d2ce7cebbb7f7528abd: Status 404 returned error can't find the container with id 503410b054f4f54772ebf1b2e056b38f029a173cf83b9d2ce7cebbb7f7528abd Oct 30 06:34:41.471667 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.471641 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt"] Oct 30 06:34:41.474084 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.474053 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:41.474156 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.474119 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:41.474156 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.474149 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:34:41.671473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:41.671421 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:42.382802 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:42.382758 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" event={"ID":"1f462dfd-9c1e-451c-9a45-07a936cccb60","Type":"ContainerStarted","Data":"567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2"} Oct 30 06:34:42.383229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:42.383184 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" event={"ID":"1f462dfd-9c1e-451c-9a45-07a936cccb60","Type":"ContainerStarted","Data":"6e48121dfac0c41d89c2669635decd1dfa1cca6ed51a7e4f417c0d3f25c3e1ac"} Oct 30 06:34:42.391698 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:42.391662 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" event={"ID":"dd098f33-cfd7-4505-8abd-0100edc4feef","Type":"ContainerStarted","Data":"e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7"} Oct 30 06:34:42.394209 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:42.394182 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" event={"ID":"dd098f33-cfd7-4505-8abd-0100edc4feef","Type":"ContainerStarted","Data":"503410b054f4f54772ebf1b2e056b38f029a173cf83b9d2ce7cebbb7f7528abd"} Oct 30 06:34:42.444003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:42.443934 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" podStartSLOduration=2.443908188 podStartE2EDuration="2.443908188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:42.43072625 +0000 UTC m=+2491.237247999" watchObservedRunningTime="2026-10-30 06:34:42.443908188 +0000 UTC m=+2491.250429940" Oct 30 06:34:42.518424 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:42.494292 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" podStartSLOduration=2.494267509 podStartE2EDuration="2.494267509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:42.478865938 +0000 UTC m=+2491.285387723" watchObservedRunningTime="2026-10-30 06:34:42.494267509 +0000 UTC m=+2491.300789261" Oct 30 06:34:42.713326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:42.695446 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:34:42.713326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:42.695518 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 06:34:43.084246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:43.084157 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:43.084675 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:43.084651 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:34:43.084831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:43.084809 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 06:34:43.140655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:43.139372 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:43.140858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:43.140644 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:34:43.140858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:43.140695 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 06:34:43.671774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:43.671737 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:34:43.672405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:43.672372 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 06:34:44.079338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:44.078712 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:34:44.079338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:44.078774 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 06:34:44.143224 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:44.142710 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:34:44.143224 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:44.142758 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 06:34:44.691335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:44.684888 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:34:44.691335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:44.684947 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 06:34:45.083392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:45.083124 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:34:45.083392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:45.083182 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 06:34:45.157327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:45.144737 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:34:45.157327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:45.144800 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 06:34:45.204462 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:45.204412 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" podUID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:34:45.690332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:45.688731 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:34:45.690332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:45.688793 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 06:34:46.091389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:46.087735 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:34:46.091389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:46.087804 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 06:34:46.138821 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:46.138788 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:34:46.139135 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:46.139111 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 06:34:46.678325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:46.676002 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:34:46.678325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:46.676061 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 06:34:47.083324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:47.082396 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:34:47.083324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:47.082455 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 06:34:47.157440 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:47.152875 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:34:47.157440 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:47.152934 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 06:34:47.682325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:47.681693 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 06:34:47.682325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:47.681756 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 06:34:48.082139 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:48.081685 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:34:48.082139 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:48.081751 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 06:34:48.143207 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:48.143157 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:34:48.143401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:48.143246 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 06:34:48.886195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:48.886153 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:48.886997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:48.886965 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:48.888078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:48.888053 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:34:49.083092 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:49.083055 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:49.083537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:49.083519 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:49.084364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:49.084343 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:34:49.170666 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:49.170592 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:49.171332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:49.171276 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:34:49.172743 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:34:49.172724 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:35:00.372609 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.372575 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh"] Oct 30 06:35:00.378682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.378658 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.393803 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.393776 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a88d7d-openshift-default-dockercfg-s6dkq\"" Oct 30 06:35:00.394939 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.394914 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.395078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.394955 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.395146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.395082 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f20b2ce-d632-4342-a6b0-c849502f09a0-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.395212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.395190 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.395254 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.395242 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27w57\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-kube-api-access-27w57\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.395319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.395289 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.395460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.395439 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.395558 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.395543 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.395629 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.395616 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.412001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.411974 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh"] Oct 30 06:35:00.496775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.496742 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.496930 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.496787 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.496930 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.496839 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.496930 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.496859 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.496928 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f20b2ce-d632-4342-a6b0-c849502f09a0-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.496990 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.497042 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27w57\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-kube-api-access-27w57\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.497106 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.497175 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.497229 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.497283 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497535 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.497499 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.497640 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.497828 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.497806 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f20b2ce-d632-4342-a6b0-c849502f09a0-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.499335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.499280 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.499506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.499487 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.519664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.519631 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.519774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.519723 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27w57\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-kube-api-access-27w57\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:00.694102 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:00.694076 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:01.014998 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:35:01.014961 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f20b2ce_d632_4342_a6b0_c849502f09a0.slice/crio-ff83fff4f3110903e55367058b9306b32e179db7c2227a999c877445d8989094 WatchSource:0}: Error finding container ff83fff4f3110903e55367058b9306b32e179db7c2227a999c877445d8989094: Status 404 returned error can't find the container with id ff83fff4f3110903e55367058b9306b32e179db7c2227a999c877445d8989094 Oct 30 06:35:01.016180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:01.016160 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh"] Oct 30 06:35:01.023650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:01.018640 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:01.023650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:01.018794 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:01.023650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:01.018885 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:01.809648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:01.809609 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" event={"ID":"1f20b2ce-d632-4342-a6b0-c849502f09a0","Type":"ContainerStarted","Data":"6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16"} Oct 30 06:35:01.810034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:01.809669 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" event={"ID":"1f20b2ce-d632-4342-a6b0-c849502f09a0","Type":"ContainerStarted","Data":"ff83fff4f3110903e55367058b9306b32e179db7c2227a999c877445d8989094"} Oct 30 06:35:01.887326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:01.884430 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" podStartSLOduration=1.884410641 podStartE2EDuration="1.884410641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:01.86807604 +0000 UTC m=+2510.674597789" watchObservedRunningTime="2026-10-30 06:35:01.884410641 +0000 UTC m=+2510.690932390" Oct 30 06:35:02.694419 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:02.694386 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:02.696493 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:02.696453 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 06:35:02.696920 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:02.696503 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 06:35:03.697245 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:03.696734 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 06:35:03.697245 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:03.696836 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 06:35:03.849323 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:35:03.843070 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:35:04.694662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:04.694629 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 06:35:04.694855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:04.694692 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 06:35:05.696326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:05.695829 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 06:35:05.696326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:05.695893 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 06:35:06.695066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:06.695034 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 06:35:06.695330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:06.695286 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 06:35:07.699228 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:07.699197 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:07.699729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:07.699516 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:07.700216 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:07.700198 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:35:10.112394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.112359 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65"] Oct 30 06:35:10.116897 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.116874 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.146883 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.146857 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-69f3fa-openshift-default-dockercfg-ls84g\"" Oct 30 06:35:10.150807 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.150777 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65"] Oct 30 06:35:10.245815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.245779 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.245989 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.245821 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66v2\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-kube-api-access-b66v2\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.245989 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.245923 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.245989 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.245962 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.246124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.246065 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.246124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.246096 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.246232 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.246129 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.246275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.246246 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.246352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.246275 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.347631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.347595 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.347837 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.347644 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66v2\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-kube-api-access-b66v2\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.347837 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.347701 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.347837 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.347742 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.347837 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.347807 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.348085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.347850 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.348085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.348048 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.348211 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.348090 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.348211 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.348090 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.348336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.348166 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.348336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.348292 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.348483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.348383 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.348483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.348464 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.348633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.348609 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.350588 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.350560 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.350803 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.350784 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.377245 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.377169 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.377551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.377527 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66v2\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-kube-api-access-b66v2\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.432005 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.431979 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:10.751358 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:35:10.751328 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dca7a50_af82_4fc2_8d40_846f77af3fc0.slice/crio-6bc6166d1571d56bf5dd946aff180edeb2fbaefe64b578e3ac4d95b48ca85de4 WatchSource:0}: Error finding container 6bc6166d1571d56bf5dd946aff180edeb2fbaefe64b578e3ac4d95b48ca85de4: Status 404 returned error can't find the container with id 6bc6166d1571d56bf5dd946aff180edeb2fbaefe64b578e3ac4d95b48ca85de4 Oct 30 06:35:10.753434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.753336 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65"] Oct 30 06:35:10.754593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.754559 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:10.754667 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.754628 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:10.754667 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.754658 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:10.926493 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.926443 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" event={"ID":"2dca7a50-af82-4fc2-8d40-846f77af3fc0","Type":"ContainerStarted","Data":"3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5"} Oct 30 06:35:10.926672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.926509 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" event={"ID":"2dca7a50-af82-4fc2-8d40-846f77af3fc0","Type":"ContainerStarted","Data":"6bc6166d1571d56bf5dd946aff180edeb2fbaefe64b578e3ac4d95b48ca85de4"} Oct 30 06:35:10.987825 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:10.987762 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" podStartSLOduration=0.987746393 podStartE2EDuration="987.746393ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:10.972737156 +0000 UTC m=+2519.779258909" watchObservedRunningTime="2026-10-30 06:35:10.987746393 +0000 UTC m=+2519.794268142" Oct 30 06:35:11.432805 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:11.432774 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:11.434210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:11.434182 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 06:35:11.434361 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:11.434240 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 06:35:12.433271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:12.433235 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 06:35:12.433748 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:12.433341 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 06:35:13.433046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:13.433011 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 06:35:13.433227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:13.433078 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 06:35:14.432934 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:14.432897 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 06:35:14.433546 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:14.432957 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 06:35:15.433182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:15.433145 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 06:35:15.433631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:15.433207 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 06:35:16.432763 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:16.432729 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 06:35:16.432955 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:16.432775 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 06:35:17.438170 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:17.438139 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:17.438583 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:17.438552 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:17.439170 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:17.439151 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:35:33.825259 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:35:33.825182 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:35:37.935458 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:37.935427 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:35:37.935896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:37.935472 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:35:40.233836 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:40.233803 4431 generic.go:356] "Generic (PLEG): container finished" podID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerID="abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4" exitCode=0 Oct 30 06:35:40.234280 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:40.233875 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" event={"ID":"253e2111-a9ac-4ddf-a3f8-6e86965c99aa","Type":"ContainerDied","Data":"abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4"} Oct 30 06:35:40.234280 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:40.234234 4431 scope.go:122] "RemoveContainer" containerID="abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4" Oct 30 06:35:40.235854 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:40.235822 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:40.235978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:40.235907 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:40.235978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:40.235959 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:41.025007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:41.024976 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7"] Oct 30 06:35:41.047097 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:41.047058 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8"] Oct 30 06:35:41.298097 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:41.297997 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" event={"ID":"253e2111-a9ac-4ddf-a3f8-6e86965c99aa","Type":"ContainerStarted","Data":"ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee"} Oct 30 06:35:41.482331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:41.481704 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:35:41.482331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:41.481833 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:35:41.482331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:41.481884 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 06:35:42.482439 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:42.482402 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:35:42.483018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:42.482984 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 06:35:43.482594 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:43.477881 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:35:43.482594 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:43.477945 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 06:35:44.484513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:44.484467 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:35:44.485038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:44.484543 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 06:35:45.137774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.137741 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4"] Oct 30 06:35:45.144076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.144047 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.182373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.162195 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4"] Oct 30 06:35:45.207330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.182938 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"routing-overlay\"" Oct 30 06:35:45.207652 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.183702 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"agc-cred-f77c5c955e8f\"" Oct 30 06:35:45.207846 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.183791 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"payload-processing-external-model-e2e--c777bb0f-dockercfg-tkbc2\"" Oct 30 06:35:45.207954 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.183862 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"openshift-service-ca.crt\"" Oct 30 06:35:45.208044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.183911 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6d1778\"" Oct 30 06:35:45.208135 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.183982 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6d1778\"/\"kube-root-ca.crt\"" Oct 30 06:35:45.271325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.270537 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.271325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.270677 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.271325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.270744 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlmm\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-kube-api-access-lzlmm\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.271325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.270885 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.407375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.401658 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.407375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.401742 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlmm\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-kube-api-access-lzlmm\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.407375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.401875 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.407375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.402062 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.442325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.435617 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.442325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.436547 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.480326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.477605 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:35:45.480326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.477672 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 06:35:45.480326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.477763 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.520329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.488466 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlmm\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-kube-api-access-lzlmm\") pod \"payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.603409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.603366 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n"] Oct 30 06:35:45.612881 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.612852 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n"] Oct 30 06:35:45.613173 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.613156 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.715002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.714968 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.715177 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.715039 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.715177 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.715151 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.715274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.715251 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.715386 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.715371 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.715451 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.715435 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.715494 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.715487 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.715537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.715525 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.715662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.715648 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmv6\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-kube-api-access-bzmv6\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.784560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.784522 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:45.817079 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817046 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817204 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817124 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817204 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817159 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817276 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817205 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817419 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817319 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmv6\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-kube-api-access-bzmv6\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817419 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817370 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817423 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817490 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817517 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817726 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817600 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.817942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817918 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.818052 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.817861 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.818253 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.818232 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.818549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.818527 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.820135 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.820108 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.821522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.821496 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.841711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.841688 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmv6\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-kube-api-access-bzmv6\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.841830 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.841685 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:45.931016 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:45.930981 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:46.028316 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.028203 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" podUID="6208c4e3-03cf-4149-bdad-57b37eb2b759" containerName="payload-processing" containerID="cri-o://972c2b089a145348a713a5824e37cbf4ea3a46301fbbc1461ba506d17cef702c" gracePeriod=25 Oct 30 06:35:46.048316 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.047706 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" podUID="3f801576-e5e0-4065-bfc7-bb74f7c3e839" containerName="payload-pre-processing" containerID="cri-o://8c6b0aa5845dcc8e97566614f913e23cc5a5fec7ce2a4f89c4a03a48b60f2869" gracePeriod=25 Oct 30 06:35:46.097019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.096990 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4"] Oct 30 06:35:46.148563 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:35:46.148531 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551737c6_162f_4c3b_b2ae_dba44f39e5eb.slice/crio-3a42ff222aa7e12bf3445ed9ba3f12b88e8d596d6657d908a1a173525c446337 WatchSource:0}: Error finding container 3a42ff222aa7e12bf3445ed9ba3f12b88e8d596d6657d908a1a173525c446337: Status 404 returned error can't find the container with id 3a42ff222aa7e12bf3445ed9ba3f12b88e8d596d6657d908a1a173525c446337 Oct 30 06:35:46.304144 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:35:46.304104 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061fb1ec_a33d_4c2a_adde_ba8a3674b9af.slice/crio-34d92ecd8acb5b142f241df66bd8d4972fae6af98f7cf5cc659c1cd2498fe909 WatchSource:0}: Error finding container 34d92ecd8acb5b142f241df66bd8d4972fae6af98f7cf5cc659c1cd2498fe909: Status 404 returned error can't find the container with id 34d92ecd8acb5b142f241df66bd8d4972fae6af98f7cf5cc659c1cd2498fe909 Oct 30 06:35:46.313543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.312217 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:46.313543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.312333 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:46.313543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.312385 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:46.322595 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.321559 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n"] Oct 30 06:35:46.502576 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.492412 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:35:46.502576 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.492490 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 06:35:46.507664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.507629 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" event={"ID":"061fb1ec-a33d-4c2a-adde-ba8a3674b9af","Type":"ContainerStarted","Data":"34d92ecd8acb5b142f241df66bd8d4972fae6af98f7cf5cc659c1cd2498fe909"} Oct 30 06:35:46.509267 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.509241 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" event={"ID":"551737c6-162f-4c3b-b2ae-dba44f39e5eb","Type":"ContainerStarted","Data":"cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547"} Oct 30 06:35:46.509434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.509290 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" event={"ID":"551737c6-162f-4c3b-b2ae-dba44f39e5eb","Type":"ContainerStarted","Data":"3a42ff222aa7e12bf3445ed9ba3f12b88e8d596d6657d908a1a173525c446337"} Oct 30 06:35:46.510253 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.510234 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:46.512238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.512148 4431 generic.go:356] "Generic (PLEG): container finished" podID="3f801576-e5e0-4065-bfc7-bb74f7c3e839" containerID="8c6b0aa5845dcc8e97566614f913e23cc5a5fec7ce2a4f89c4a03a48b60f2869" exitCode=0 Oct 30 06:35:46.512238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.512215 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" event={"ID":"3f801576-e5e0-4065-bfc7-bb74f7c3e839","Type":"ContainerDied","Data":"8c6b0aa5845dcc8e97566614f913e23cc5a5fec7ce2a4f89c4a03a48b60f2869"} Oct 30 06:35:46.513557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.513537 4431 generic.go:356] "Generic (PLEG): container finished" podID="6208c4e3-03cf-4149-bdad-57b37eb2b759" containerID="972c2b089a145348a713a5824e37cbf4ea3a46301fbbc1461ba506d17cef702c" exitCode=0 Oct 30 06:35:46.513727 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.513714 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" event={"ID":"6208c4e3-03cf-4149-bdad-57b37eb2b759","Type":"ContainerDied","Data":"972c2b089a145348a713a5824e37cbf4ea3a46301fbbc1461ba506d17cef702c"} Oct 30 06:35:46.513799 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.513746 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" event={"ID":"6208c4e3-03cf-4149-bdad-57b37eb2b759","Type":"ContainerDied","Data":"8e5d90cf8355d71a2594980c610fb0684ea4eca1db8b96e08a21d17c45ea82b1"} Oct 30 06:35:46.513799 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.513764 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e5d90cf8355d71a2594980c610fb0684ea4eca1db8b96e08a21d17c45ea82b1" Oct 30 06:35:46.553348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.552985 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" podStartSLOduration=1.552968902 podStartE2EDuration="1.552968902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:46.549878453 +0000 UTC m=+2555.356400201" watchObservedRunningTime="2026-10-30 06:35:46.552968902 +0000 UTC m=+2555.359490651" Oct 30 06:35:46.573523 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.573499 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:35:46.599749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.599725 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:35:46.798014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.768516 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6208c4e3-03cf-4149-bdad-57b37eb2b759-kube-api-access-2kxwk\" (UniqueName: \"kubernetes.io/projected/6208c4e3-03cf-4149-bdad-57b37eb2b759-kube-api-access-2kxwk\") pod \"6208c4e3-03cf-4149-bdad-57b37eb2b759\" (UID: \"6208c4e3-03cf-4149-bdad-57b37eb2b759\") " Oct 30 06:35:46.798014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.768642 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f801576-e5e0-4065-bfc7-bb74f7c3e839-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f801576-e5e0-4065-bfc7-bb74f7c3e839-plugins-config-volume\") pod \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\" (UID: \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\") " Oct 30 06:35:46.798014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.768756 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6208c4e3-03cf-4149-bdad-57b37eb2b759-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6208c4e3-03cf-4149-bdad-57b37eb2b759-plugins-config-volume\") pod \"6208c4e3-03cf-4149-bdad-57b37eb2b759\" (UID: \"6208c4e3-03cf-4149-bdad-57b37eb2b759\") " Oct 30 06:35:46.798014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.768797 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f801576-e5e0-4065-bfc7-bb74f7c3e839-kube-api-access-lpzkl\" (UniqueName: \"kubernetes.io/projected/3f801576-e5e0-4065-bfc7-bb74f7c3e839-kube-api-access-lpzkl\") pod \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\" (UID: \"3f801576-e5e0-4065-bfc7-bb74f7c3e839\") " Oct 30 06:35:46.798014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.782506 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f801576-e5e0-4065-bfc7-bb74f7c3e839-plugins-config-volume" pod "3f801576-e5e0-4065-bfc7-bb74f7c3e839" (UID: "3f801576-e5e0-4065-bfc7-bb74f7c3e839"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:46.798014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.797227 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6208c4e3-03cf-4149-bdad-57b37eb2b759-plugins-config-volume" pod "6208c4e3-03cf-4149-bdad-57b37eb2b759" (UID: "6208c4e3-03cf-4149-bdad-57b37eb2b759"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:46.799000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.798979 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6208c4e3-03cf-4149-bdad-57b37eb2b759-kube-api-access-2kxwk" pod "6208c4e3-03cf-4149-bdad-57b37eb2b759" (UID: "6208c4e3-03cf-4149-bdad-57b37eb2b759"). InnerVolumeSpecName "kube-api-access-2kxwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:46.807458 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.807394 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f801576-e5e0-4065-bfc7-bb74f7c3e839-kube-api-access-lpzkl" pod "3f801576-e5e0-4065-bfc7-bb74f7c3e839" (UID: "3f801576-e5e0-4065-bfc7-bb74f7c3e839"). InnerVolumeSpecName "kube-api-access-lpzkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:46.874336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.873607 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kxwk\" (UniqueName: \"kubernetes.io/projected/6208c4e3-03cf-4149-bdad-57b37eb2b759-kube-api-access-2kxwk\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:46.874336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.873644 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f801576-e5e0-4065-bfc7-bb74f7c3e839-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:46.874336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.873659 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6208c4e3-03cf-4149-bdad-57b37eb2b759-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:46.874336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:46.873674 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpzkl\" (UniqueName: \"kubernetes.io/projected/3f801576-e5e0-4065-bfc7-bb74f7c3e839-kube-api-access-lpzkl\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.527370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.527333 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" event={"ID":"061fb1ec-a33d-4c2a-adde-ba8a3674b9af","Type":"ContainerStarted","Data":"58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36"} Oct 30 06:35:47.528981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.528955 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" Oct 30 06:35:47.529121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.528972 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8" event={"ID":"3f801576-e5e0-4065-bfc7-bb74f7c3e839","Type":"ContainerDied","Data":"4a163df5abc5971968a3d8a1f6e2028d1f38e13e1b776ad6ed428213a425629e"} Oct 30 06:35:47.529121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.529037 4431 scope.go:122] "RemoveContainer" containerID="8c6b0aa5845dcc8e97566614f913e23cc5a5fec7ce2a4f89c4a03a48b60f2869" Oct 30 06:35:47.529121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.529107 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7" Oct 30 06:35:47.617767 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.617428 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:35:47.617767 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.617748 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:35:47.619887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.619117 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:35:47.627846 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.627792 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" podStartSLOduration=2.627776012 podStartE2EDuration="2.627776012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:47.621392726 +0000 UTC m=+2556.427914475" watchObservedRunningTime="2026-10-30 06:35:47.627776012 +0000 UTC m=+2556.434297760" Oct 30 06:35:47.727229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.727193 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8"] Oct 30 06:35:47.734013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.733979 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a88d7d-784b755d79-qvjj8"] Oct 30 06:35:47.790438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.790398 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7"] Oct 30 06:35:47.795545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.795476 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a88d7d-7d7f79555b-gkqr7"] Oct 30 06:35:47.932532 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.932445 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:47.934575 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.934549 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 06:35:47.934713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:47.934592 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 06:35:48.932611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:48.932583 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 06:35:48.932992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:48.932646 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 06:35:49.343186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:49.343150 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f801576-e5e0-4065-bfc7-bb74f7c3e839" path="/var/lib/kubelet/pods/3f801576-e5e0-4065-bfc7-bb74f7c3e839/volumes" Oct 30 06:35:49.343816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:49.343797 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6208c4e3-03cf-4149-bdad-57b37eb2b759" path="/var/lib/kubelet/pods/6208c4e3-03cf-4149-bdad-57b37eb2b759/volumes" Oct 30 06:35:49.932692 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:49.932655 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 06:35:49.933057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:49.932705 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 06:35:50.932516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:50.932481 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 06:35:50.932670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:50.932534 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 06:35:51.932396 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:51.932364 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 06:35:51.932799 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:51.932438 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 06:35:52.931965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:52.931934 4431 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 06:35:52.932152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:52.931987 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 06:35:53.613798 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:53.613708 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4"] Oct 30 06:35:53.615367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:53.613960 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" containerID="cri-o://2887e42ff66885f6221dd218cdfc91cd48924ef11902fd12b36db7f8d961c61f" gracePeriod=30 Oct 30 06:35:53.937081 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:53.937006 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:53.937438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:53.937415 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:53.938063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:53.938044 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:35:55.247788 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.247756 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf"] Oct 30 06:35:55.248455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.248440 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6208c4e3-03cf-4149-bdad-57b37eb2b759" containerName="payload-processing" Oct 30 06:35:55.248518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.248456 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6208c4e3-03cf-4149-bdad-57b37eb2b759" containerName="payload-processing" Oct 30 06:35:55.248518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.248470 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3f801576-e5e0-4065-bfc7-bb74f7c3e839" containerName="payload-pre-processing" Oct 30 06:35:55.248518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.248475 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f801576-e5e0-4065-bfc7-bb74f7c3e839" containerName="payload-pre-processing" Oct 30 06:35:55.248614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.248579 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6208c4e3-03cf-4149-bdad-57b37eb2b759" containerName="payload-processing" Oct 30 06:35:55.248614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.248587 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f801576-e5e0-4065-bfc7-bb74f7c3e839" containerName="payload-pre-processing" Oct 30 06:35:55.253772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.253750 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.276683 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.276659 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf"] Oct 30 06:35:55.286514 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.286490 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8"] Oct 30 06:35:55.291170 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.291152 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.309459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.309436 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8"] Oct 30 06:35:55.324969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.324943 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.325094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.324984 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58609e0-908f-4c1a-93f8-a4cde0954868-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.325143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325093 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.325143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325120 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.325248 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325203 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.325354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325331 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.325401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325376 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.325461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325405 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.325555 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325532 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.325617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325582 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.325695 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325682 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.325768 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325753 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95nzp\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-kube-api-access-95nzp\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.325832 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325804 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj4dh\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-kube-api-access-dj4dh\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.325943 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325837 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.325943 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325863 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.325943 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325917 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.326098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.325989 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.326098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.326019 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.427105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427069 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.427251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427207 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.427341 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427279 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95nzp\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-kube-api-access-95nzp\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.427421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427339 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj4dh\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-kube-api-access-dj4dh\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.427421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427367 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.427421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427392 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.427566 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427466 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.427566 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427518 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.427566 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427553 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.427732 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427592 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.427732 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427658 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.427732 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427683 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58609e0-908f-4c1a-93f8-a4cde0954868-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.427898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427742 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.427898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427770 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.427898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427824 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.427898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427842 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.427898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427862 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.428155 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427911 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.428155 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.427942 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.428155 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.428047 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.428550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.428517 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.428550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.428524 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.428806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.428785 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.428883 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.428810 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.428883 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.428843 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.429023 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.428994 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58609e0-908f-4c1a-93f8-a4cde0954868-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.429111 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.429090 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.429391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.429369 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.430377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.430357 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.430783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.430760 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.430913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.430894 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.431339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.431289 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.516592 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.512519 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj4dh\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-kube-api-access-dj4dh\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.516592 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.513022 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.524890 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.523468 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.524890 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.523564 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95nzp\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-kube-api-access-95nzp\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.569120 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.569091 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:55.611670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.611643 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:55.885358 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:35:55.885326 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58609e0_908f_4c1a_93f8_a4cde0954868.slice/crio-15f1afe9c7d776ddae2a9f3c53bcdc39da5e1beebcf2edbd9424194e8ecd4aae WatchSource:0}: Error finding container 15f1afe9c7d776ddae2a9f3c53bcdc39da5e1beebcf2edbd9424194e8ecd4aae: Status 404 returned error can't find the container with id 15f1afe9c7d776ddae2a9f3c53bcdc39da5e1beebcf2edbd9424194e8ecd4aae Oct 30 06:35:55.886535 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.886506 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf"] Oct 30 06:35:55.888493 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.888456 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:55.888595 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.888542 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:55.888595 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.888587 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:55.936779 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:35:55.936751 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ee2fcd_1dfb_47d6_b7aa_395160feb05a.slice/crio-a5cba46ce259ad43e37eaac99b7449275e709baa0813701278f60a7e5155525a WatchSource:0}: Error finding container a5cba46ce259ad43e37eaac99b7449275e709baa0813701278f60a7e5155525a: Status 404 returned error can't find the container with id a5cba46ce259ad43e37eaac99b7449275e709baa0813701278f60a7e5155525a Oct 30 06:35:55.938201 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.938176 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8"] Oct 30 06:35:55.940674 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.940641 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:55.940768 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.940716 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:55.940768 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:55.940759 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:35:56.638625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:56.638585 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" event={"ID":"12ee2fcd-1dfb-47d6-b7aa-395160feb05a","Type":"ContainerStarted","Data":"cfac97b48f5fbb0a07ea5128676837c1a261d09b6e7fc834f8d288e03274cd64"} Oct 30 06:35:56.639064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:56.638647 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" event={"ID":"12ee2fcd-1dfb-47d6-b7aa-395160feb05a","Type":"ContainerStarted","Data":"a5cba46ce259ad43e37eaac99b7449275e709baa0813701278f60a7e5155525a"} Oct 30 06:35:56.640332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:56.640271 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" event={"ID":"f58609e0-908f-4c1a-93f8-a4cde0954868","Type":"ContainerStarted","Data":"9d0e8c31511d1c4152624a3a1da8e60d578199daa43020cbbace6ead5d24f63c"} Oct 30 06:35:56.640444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:56.640342 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" event={"ID":"f58609e0-908f-4c1a-93f8-a4cde0954868","Type":"ContainerStarted","Data":"15f1afe9c7d776ddae2a9f3c53bcdc39da5e1beebcf2edbd9424194e8ecd4aae"} Oct 30 06:35:56.789330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:56.788621 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" podStartSLOduration=1.788599365 podStartE2EDuration="1.788599365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:56.708918959 +0000 UTC m=+2565.515440708" watchObservedRunningTime="2026-10-30 06:35:56.788599365 +0000 UTC m=+2565.595121114" Oct 30 06:35:57.571324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:57.570650 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:35:57.594324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:57.593460 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:35:57.594324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:57.593528 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 06:35:57.614328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:57.612983 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:35:57.614840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:57.614819 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:35:57.614891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:57.614858 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 06:35:58.539913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:58.539876 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:35:58.582326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:58.578717 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:35:58.582326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:58.578773 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 06:35:58.618403 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:58.615646 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:35:58.618403 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:58.615698 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 06:35:58.620365 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:58.620184 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" podStartSLOduration=3.620163157 podStartE2EDuration="3.620163157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:56.783897066 +0000 UTC m=+2565.590418851" watchObservedRunningTime="2026-10-30 06:35:58.620163157 +0000 UTC m=+2567.426684909" Oct 30 06:35:58.657069 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:58.657045 4431 generic.go:356] "Generic (PLEG): container finished" podID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerID="2887e42ff66885f6221dd218cdfc91cd48924ef11902fd12b36db7f8d961c61f" exitCode=0 Oct 30 06:35:58.657188 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:58.657128 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" event={"ID":"131ee3df-7381-4d80-9ac5-5b0c104bcc01","Type":"ContainerDied","Data":"2887e42ff66885f6221dd218cdfc91cd48924ef11902fd12b36db7f8d961c61f"} Oct 30 06:35:59.551645 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.551619 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:35:59.591335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.583878 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:35:59.591335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.583935 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 06:35:59.612645 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.612611 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:35:59.612787 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.612653 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 06:35:59.648950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.648914 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-envoy\") pod \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " Oct 30 06:35:59.649101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.648962 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-socket\") pod \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " Oct 30 06:35:59.649101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.649066 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-credential-socket\") pod \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " Oct 30 06:35:59.649222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.649139 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-token\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-token\") pod \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " Oct 30 06:35:59.649222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.649183 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-data\") pod \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " Oct 30 06:35:59.649356 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.649341 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-kube-api-access-gs6gw\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-kube-api-access-gs6gw\") pod \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " Oct 30 06:35:59.650510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.649464 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-certs\") pod \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " Oct 30 06:35:59.650510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.649506 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-podinfo\") pod \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " Oct 30 06:35:59.650510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.649586 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istiod-ca-cert\") pod \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\" (UID: \"131ee3df-7381-4d80-9ac5-5b0c104bcc01\") " Oct 30 06:35:59.650735 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.650629 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-socket" pod "131ee3df-7381-4d80-9ac5-5b0c104bcc01" (UID: "131ee3df-7381-4d80-9ac5-5b0c104bcc01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.650735 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.650642 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-data" pod "131ee3df-7381-4d80-9ac5-5b0c104bcc01" (UID: "131ee3df-7381-4d80-9ac5-5b0c104bcc01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.650735 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.650660 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-credential-socket" pod "131ee3df-7381-4d80-9ac5-5b0c104bcc01" (UID: "131ee3df-7381-4d80-9ac5-5b0c104bcc01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.650935 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.650755 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-certs" pod "131ee3df-7381-4d80-9ac5-5b0c104bcc01" (UID: "131ee3df-7381-4d80-9ac5-5b0c104bcc01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.651151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.651119 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istiod-ca-cert" pod "131ee3df-7381-4d80-9ac5-5b0c104bcc01" (UID: "131ee3df-7381-4d80-9ac5-5b0c104bcc01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:59.653058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.653033 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-token" pod "131ee3df-7381-4d80-9ac5-5b0c104bcc01" (UID: "131ee3df-7381-4d80-9ac5-5b0c104bcc01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:59.653290 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.653272 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-podinfo" pod "131ee3df-7381-4d80-9ac5-5b0c104bcc01" (UID: "131ee3df-7381-4d80-9ac5-5b0c104bcc01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:35:59.653367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.653286 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-kube-api-access-gs6gw" pod "131ee3df-7381-4d80-9ac5-5b0c104bcc01" (UID: "131ee3df-7381-4d80-9ac5-5b0c104bcc01"). InnerVolumeSpecName "kube-api-access-gs6gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:59.653866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.653845 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-envoy" pod "131ee3df-7381-4d80-9ac5-5b0c104bcc01" (UID: "131ee3df-7381-4d80-9ac5-5b0c104bcc01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.725227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.725202 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" Oct 30 06:35:59.725393 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.725284 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4" event={"ID":"131ee3df-7381-4d80-9ac5-5b0c104bcc01","Type":"ContainerDied","Data":"90417fcf120b0aad87a1f54d5aa7a20e99ceb31d445c9408a9e85f0427116dc2"} Oct 30 06:35:59.725393 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.725386 4431 scope.go:122] "RemoveContainer" containerID="2887e42ff66885f6221dd218cdfc91cd48924ef11902fd12b36db7f8d961c61f" Oct 30 06:35:59.755250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.755217 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs6gw\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-kube-api-access-gs6gw\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.755250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.755255 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.755250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.755265 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.755562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.755274 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.755562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.755286 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.755562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.755317 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.755562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.755330 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.755562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.755339 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.755562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.755347 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131ee3df-7381-4d80-9ac5-5b0c104bcc01-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.823347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.820603 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4"] Oct 30 06:35:59.823347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:35:59.823213 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-fgvg4"] Oct 30 06:36:00.588326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:00.587691 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:36:00.588326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:00.587751 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 06:36:00.612969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:00.612934 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:36:00.613128 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:00.613005 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 06:36:01.339977 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:01.339942 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" path="/var/lib/kubelet/pods/131ee3df-7381-4d80-9ac5-5b0c104bcc01/volumes" Oct 30 06:36:01.571786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:01.571748 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:36:01.572045 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:01.572014 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 06:36:01.612992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:01.612894 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:36:01.612992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:01.612954 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 06:36:02.574405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:02.574371 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:36:02.574885 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:02.574861 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:36:02.575548 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:02.575527 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:36:02.617013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:02.616990 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:36:02.617339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:02.617266 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:36:02.617975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:02.617954 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:36:03.702282 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:36:03.702246 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:36:31.428784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:31.428752 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n"] Oct 30 06:36:31.429432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:31.429413 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh"] Oct 30 06:36:31.429829 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:31.429795 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" containerID="cri-o://58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36" gracePeriod=30 Oct 30 06:36:31.430124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:31.430097 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerName="istio-proxy" containerID="cri-o://6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16" gracePeriod=30 Oct 30 06:36:33.701544 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:36:33.701506 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:36:36.864558 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.864530 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:36:36.919233 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.919207 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:36:36.940472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.940441 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-kube-api-access-27w57\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-kube-api-access-27w57\") pod \"1f20b2ce-d632-4342-a6b0-c849502f09a0\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " Oct 30 06:36:36.940614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.940490 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-token\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-token\") pod \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " Oct 30 06:36:36.940688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.940610 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-token\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-token\") pod \"1f20b2ce-d632-4342-a6b0-c849502f09a0\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " Oct 30 06:36:36.940688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.940676 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-socket\") pod \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " Oct 30 06:36:36.940810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.940712 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-socket\") pod \"1f20b2ce-d632-4342-a6b0-c849502f09a0\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " Oct 30 06:36:36.940810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.940760 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f20b2ce-d632-4342-a6b0-c849502f09a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f20b2ce-d632-4342-a6b0-c849502f09a0-istiod-ca-cert\") pod \"1f20b2ce-d632-4342-a6b0-c849502f09a0\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " Oct 30 06:36:36.940932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.940843 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-credential-socket\") pod \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " Oct 30 06:36:36.941017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.940964 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-data\") pod \"1f20b2ce-d632-4342-a6b0-c849502f09a0\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " Oct 30 06:36:36.941017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941004 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-envoy\") pod \"1f20b2ce-d632-4342-a6b0-c849502f09a0\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " Oct 30 06:36:36.941135 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941031 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-data\") pod \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " Oct 30 06:36:36.941135 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941083 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-kube-api-access-bzmv6\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-kube-api-access-bzmv6\") pod \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " Oct 30 06:36:36.941255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941137 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-certs\") pod \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " Oct 30 06:36:36.941255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941179 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-credential-socket\") pod \"1f20b2ce-d632-4342-a6b0-c849502f09a0\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " Oct 30 06:36:36.941255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941223 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-podinfo\") pod \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " Oct 30 06:36:36.941467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941274 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-podinfo\") pod \"1f20b2ce-d632-4342-a6b0-c849502f09a0\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " Oct 30 06:36:36.941467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941363 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istiod-ca-cert\") pod \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " Oct 30 06:36:36.941467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941392 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-envoy\") pod \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\" (UID: \"061fb1ec-a33d-4c2a-adde-ba8a3674b9af\") " Oct 30 06:36:36.941467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.941426 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-certs\") pod \"1f20b2ce-d632-4342-a6b0-c849502f09a0\" (UID: \"1f20b2ce-d632-4342-a6b0-c849502f09a0\") " Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.942479 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-certs" pod "1f20b2ce-d632-4342-a6b0-c849502f09a0" (UID: "1f20b2ce-d632-4342-a6b0-c849502f09a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.942543 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-credential-socket" pod "061fb1ec-a33d-4c2a-adde-ba8a3674b9af" (UID: "061fb1ec-a33d-4c2a-adde-ba8a3674b9af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.942832 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-credential-socket" pod "1f20b2ce-d632-4342-a6b0-c849502f09a0" (UID: "1f20b2ce-d632-4342-a6b0-c849502f09a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.942408 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f20b2ce-d632-4342-a6b0-c849502f09a0-istiod-ca-cert" pod "1f20b2ce-d632-4342-a6b0-c849502f09a0" (UID: "1f20b2ce-d632-4342-a6b0-c849502f09a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.943149 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-data" pod "061fb1ec-a33d-4c2a-adde-ba8a3674b9af" (UID: "061fb1ec-a33d-4c2a-adde-ba8a3674b9af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.943207 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-token" pod "1f20b2ce-d632-4342-a6b0-c849502f09a0" (UID: "1f20b2ce-d632-4342-a6b0-c849502f09a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.943254 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-token" pod "061fb1ec-a33d-4c2a-adde-ba8a3674b9af" (UID: "061fb1ec-a33d-4c2a-adde-ba8a3674b9af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.943544 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istiod-ca-cert" pod "061fb1ec-a33d-4c2a-adde-ba8a3674b9af" (UID: "061fb1ec-a33d-4c2a-adde-ba8a3674b9af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.943548 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-kube-api-access-27w57" pod "1f20b2ce-d632-4342-a6b0-c849502f09a0" (UID: "1f20b2ce-d632-4342-a6b0-c849502f09a0"). InnerVolumeSpecName "kube-api-access-27w57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.943791 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-socket" pod "1f20b2ce-d632-4342-a6b0-c849502f09a0" (UID: "1f20b2ce-d632-4342-a6b0-c849502f09a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.943835 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-socket" pod "061fb1ec-a33d-4c2a-adde-ba8a3674b9af" (UID: "061fb1ec-a33d-4c2a-adde-ba8a3674b9af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.943909 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-certs" pod "061fb1ec-a33d-4c2a-adde-ba8a3674b9af" (UID: "061fb1ec-a33d-4c2a-adde-ba8a3674b9af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.943967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.943970 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-data" pod "1f20b2ce-d632-4342-a6b0-c849502f09a0" (UID: "1f20b2ce-d632-4342-a6b0-c849502f09a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.944944 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.944855 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-podinfo" pod "061fb1ec-a33d-4c2a-adde-ba8a3674b9af" (UID: "061fb1ec-a33d-4c2a-adde-ba8a3674b9af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:36:36.945434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.945397 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-kube-api-access-bzmv6" pod "061fb1ec-a33d-4c2a-adde-ba8a3674b9af" (UID: "061fb1ec-a33d-4c2a-adde-ba8a3674b9af"). InnerVolumeSpecName "kube-api-access-bzmv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:36.945643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.945615 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-envoy" pod "061fb1ec-a33d-4c2a-adde-ba8a3674b9af" (UID: "061fb1ec-a33d-4c2a-adde-ba8a3674b9af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.945857 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.945839 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-podinfo" pod "1f20b2ce-d632-4342-a6b0-c849502f09a0" (UID: "1f20b2ce-d632-4342-a6b0-c849502f09a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:36:36.946447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:36.946430 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-envoy" pod "1f20b2ce-d632-4342-a6b0-c849502f09a0" (UID: "1f20b2ce-d632-4342-a6b0-c849502f09a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:37.044001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.043933 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.043958 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.043968 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.043978 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27w57\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-kube-api-access-27w57\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.043987 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.043996 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044003 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044011 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044021 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f20b2ce-d632-4342-a6b0-c849502f09a0-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044031 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044039 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044047 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044054 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044061 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzmv6\" (UniqueName: \"kubernetes.io/projected/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-kube-api-access-bzmv6\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044069 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044077 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f20b2ce-d632-4342-a6b0-c849502f09a0-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044084 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061fb1ec-a33d-4c2a-adde-ba8a3674b9af-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.044368 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.044091 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f20b2ce-d632-4342-a6b0-c849502f09a0-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.338739 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.338652 4431 generic.go:356] "Generic (PLEG): container finished" podID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerID="58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36" exitCode=0 Oct 30 06:36:37.338914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.338777 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" Oct 30 06:36:37.340265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.340199 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" event={"ID":"061fb1ec-a33d-4c2a-adde-ba8a3674b9af","Type":"ContainerDied","Data":"58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36"} Oct 30 06:36:37.340265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.340254 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n" event={"ID":"061fb1ec-a33d-4c2a-adde-ba8a3674b9af","Type":"ContainerDied","Data":"34d92ecd8acb5b142f241df66bd8d4972fae6af98f7cf5cc659c1cd2498fe909"} Oct 30 06:36:37.340475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.340285 4431 scope.go:122] "RemoveContainer" containerID="58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36" Oct 30 06:36:37.340475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.340315 4431 generic.go:356] "Generic (PLEG): container finished" podID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerID="6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16" exitCode=0 Oct 30 06:36:37.340475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.340380 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" Oct 30 06:36:37.340475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.340382 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" event={"ID":"1f20b2ce-d632-4342-a6b0-c849502f09a0","Type":"ContainerDied","Data":"6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16"} Oct 30 06:36:37.340475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.340425 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh" event={"ID":"1f20b2ce-d632-4342-a6b0-c849502f09a0","Type":"ContainerDied","Data":"ff83fff4f3110903e55367058b9306b32e179db7c2227a999c877445d8989094"} Oct 30 06:36:37.359354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.359332 4431 scope.go:122] "RemoveContainer" containerID="58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36" Oct 30 06:36:37.359636 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:36:37.359616 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36\": container with ID starting with 58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36 not found: ID does not exist" containerID="58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36" Oct 30 06:36:37.359699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.359643 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36"} err="failed to get container status \"58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36\": rpc error: code = NotFound desc = could not find container \"58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36\": container with ID starting with 58ce871ce5b89310807b511fd547f0f45e68fe1de3b08f15b50db1e54b966b36 not found: ID does not exist" Oct 30 06:36:37.359699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.359660 4431 scope.go:122] "RemoveContainer" containerID="6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16" Oct 30 06:36:37.369174 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.369157 4431 scope.go:122] "RemoveContainer" containerID="6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16" Oct 30 06:36:37.369481 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:36:37.369456 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16\": container with ID starting with 6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16 not found: ID does not exist" containerID="6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16" Oct 30 06:36:37.369564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.369488 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16"} err="failed to get container status \"6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16\": rpc error: code = NotFound desc = could not find container \"6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16\": container with ID starting with 6dd4f8217132cb78c211122a0b37fdaa5e1d05802a7038317e1ea4f559f89e16 not found: ID does not exist" Oct 30 06:36:37.409496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.409466 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh"] Oct 30 06:36:37.421580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.421559 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-sqhdh"] Oct 30 06:36:37.451464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.451443 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n"] Oct 30 06:36:37.455766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:37.455747 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-nfw5n"] Oct 30 06:36:39.339667 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:39.339633 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" path="/var/lib/kubelet/pods/061fb1ec-a33d-4c2a-adde-ba8a3674b9af/volumes" Oct 30 06:36:39.340179 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:39.340163 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" path="/var/lib/kubelet/pods/1f20b2ce-d632-4342-a6b0-c849502f09a0/volumes" Oct 30 06:36:40.737327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.736165 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r"] Oct 30 06:36:40.737995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.737762 4431 cpu_manager.go:414] "containerMap: removing container" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" Oct 30 06:36:40.737995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.737784 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" Oct 30 06:36:40.737995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.737864 4431 cpu_manager.go:414] "containerMap: removing container" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" Oct 30 06:36:40.737995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.737874 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" Oct 30 06:36:40.737995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.737887 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerName="istio-proxy" Oct 30 06:36:40.737995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.737895 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerName="istio-proxy" Oct 30 06:36:40.738367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.738031 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="061fb1ec-a33d-4c2a-adde-ba8a3674b9af" containerName="istio-proxy" Oct 30 06:36:40.738367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.738042 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f20b2ce-d632-4342-a6b0-c849502f09a0" containerName="istio-proxy" Oct 30 06:36:40.738367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.738054 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="131ee3df-7381-4d80-9ac5-5b0c104bcc01" containerName="istio-proxy" Oct 30 06:36:40.783322 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.773499 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r"] Oct 30 06:36:40.783322 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.773628 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:40.828571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.828540 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-88acce\"" Oct 30 06:36:40.840684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.829038 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-88acce-dockercfg-gnr69\"" Oct 30 06:36:40.863250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.863197 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5b9\" (UniqueName: \"kubernetes.io/projected/8a59b404-db18-4117-a038-181392be01d6-kube-api-access-sw5b9\") pod \"payload-processing-e2e-extmod-a-88acce-545647888d-gd22r\" (UID: \"8a59b404-db18-4117-a038-181392be01d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:40.869087 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.869041 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a59b404-db18-4117-a038-181392be01d6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-88acce-545647888d-gd22r\" (UID: \"8a59b404-db18-4117-a038-181392be01d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:40.971642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.971611 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a59b404-db18-4117-a038-181392be01d6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-88acce-545647888d-gd22r\" (UID: \"8a59b404-db18-4117-a038-181392be01d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:40.971841 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.971826 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5b9\" (UniqueName: \"kubernetes.io/projected/8a59b404-db18-4117-a038-181392be01d6-kube-api-access-sw5b9\") pod \"payload-processing-e2e-extmod-a-88acce-545647888d-gd22r\" (UID: \"8a59b404-db18-4117-a038-181392be01d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:40.972543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.972514 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a59b404-db18-4117-a038-181392be01d6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-88acce-545647888d-gd22r\" (UID: \"8a59b404-db18-4117-a038-181392be01d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:40.995176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:40.995118 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5b9\" (UniqueName: \"kubernetes.io/projected/8a59b404-db18-4117-a038-181392be01d6-kube-api-access-sw5b9\") pod \"payload-processing-e2e-extmod-a-88acce-545647888d-gd22r\" (UID: \"8a59b404-db18-4117-a038-181392be01d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:41.177562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:41.176970 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:41.494970 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:36:41.494936 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a59b404_db18_4117_a038_181392be01d6.slice/crio-87b8782731dc3fb0bd281b633804ab4b2a60490c2d0cdde1608168656035f56e WatchSource:0}: Error finding container 87b8782731dc3fb0bd281b633804ab4b2a60490c2d0cdde1608168656035f56e: Status 404 returned error can't find the container with id 87b8782731dc3fb0bd281b633804ab4b2a60490c2d0cdde1608168656035f56e Oct 30 06:36:41.495536 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:41.495513 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r"] Oct 30 06:36:42.522019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:42.521976 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" event={"ID":"8a59b404-db18-4117-a038-181392be01d6","Type":"ContainerStarted","Data":"1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3"} Oct 30 06:36:42.522557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:42.522080 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" event={"ID":"8a59b404-db18-4117-a038-181392be01d6","Type":"ContainerStarted","Data":"87b8782731dc3fb0bd281b633804ab4b2a60490c2d0cdde1608168656035f56e"} Oct 30 06:36:42.522557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:42.522115 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:53.534089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:53.534061 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:36:53.578068 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:36:53.578018 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" podStartSLOduration=13.578003365 podStartE2EDuration="13.578003365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:42.566536843 +0000 UTC m=+2611.373058591" watchObservedRunningTime="2026-10-30 06:36:53.578003365 +0000 UTC m=+2622.384525113" Oct 30 06:37:03.700437 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:37:03.700398 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:37:20.223230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.223199 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d"] Oct 30 06:37:20.229062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.229041 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.247195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.247171 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d"] Oct 30 06:37:20.254562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.254537 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"routing-overlay\"" Oct 30 06:37:20.255018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.254927 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"openshift-service-ca.crt\"" Oct 30 06:37:20.255018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.254992 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"kube-root-ca.crt\"" Oct 30 06:37:20.255186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.255096 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"agc-cred-19890eee2497\"" Oct 30 06:37:20.255501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.255473 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ad1661\"" Oct 30 06:37:20.255949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.255929 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ad1661\"/\"payload-processing-external-model-e2e--068888f6-dockercfg-xbbsn\"" Oct 30 06:37:20.400327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.398284 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.400327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.398665 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.400327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.398720 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.400327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.398781 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjp9\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-kube-api-access-ptjp9\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.504706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.504608 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.505020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.504991 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.505210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.505193 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjp9\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-kube-api-access-ptjp9\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.505475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.505457 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.508239 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.508212 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.510392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.510367 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.512071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.512033 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.535206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.535170 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjp9\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-kube-api-access-ptjp9\") pod \"payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:20.546320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:20.543684 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:21.086536 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:37:21.086512 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5fb24d_efda_4edc_aefe_46cef86a2c1b.slice/crio-4d355a38c8c5cf8e4df2e2addc533c8f804a8c462941111cb673b07c72a1652f WatchSource:0}: Error finding container 4d355a38c8c5cf8e4df2e2addc533c8f804a8c462941111cb673b07c72a1652f: Status 404 returned error can't find the container with id 4d355a38c8c5cf8e4df2e2addc533c8f804a8c462941111cb673b07c72a1652f Oct 30 06:37:21.088746 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:21.088720 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d"] Oct 30 06:37:22.053668 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:22.053635 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" event={"ID":"db5fb24d-efda-4edc-aefe-46cef86a2c1b","Type":"ContainerStarted","Data":"0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b"} Oct 30 06:37:22.054062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:22.053685 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" event={"ID":"db5fb24d-efda-4edc-aefe-46cef86a2c1b","Type":"ContainerStarted","Data":"4d355a38c8c5cf8e4df2e2addc533c8f804a8c462941111cb673b07c72a1652f"} Oct 30 06:37:22.054062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:22.053730 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:22.098659 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:22.098568 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" podStartSLOduration=2.098545054 podStartE2EDuration="2.098545054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:37:22.09593804 +0000 UTC m=+2650.902459789" watchObservedRunningTime="2026-10-30 06:37:22.098545054 +0000 UTC m=+2650.905066803" Oct 30 06:37:27.448127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:27.448092 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d"] Oct 30 06:37:27.451465 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:27.451442 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:32.449421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.449368 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" podUID="db5fb24d-efda-4edc-aefe-46cef86a2c1b" containerName="payload-processing" containerID="cri-o://0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b" gracePeriod=25 Oct 30 06:37:32.730596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.730567 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:32.816992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.816965 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-kube-api-access-ptjp9\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-kube-api-access-ptjp9\") pod \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " Oct 30 06:37:32.817138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.817072 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-plugins-config-volume\") pod \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " Oct 30 06:37:32.817227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.817136 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-routing-overlay-volume\") pod \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " Oct 30 06:37:32.817291 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.817240 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-provider-credentials\") pod \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\" (UID: \"db5fb24d-efda-4edc-aefe-46cef86a2c1b\") " Oct 30 06:37:32.817496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.817474 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-routing-overlay-volume" pod "db5fb24d-efda-4edc-aefe-46cef86a2c1b" (UID: "db5fb24d-efda-4edc-aefe-46cef86a2c1b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:32.817571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.817548 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-plugins-config-volume" pod "db5fb24d-efda-4edc-aefe-46cef86a2c1b" (UID: "db5fb24d-efda-4edc-aefe-46cef86a2c1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:32.817989 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.817968 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:37:32.818129 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.818114 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db5fb24d-efda-4edc-aefe-46cef86a2c1b-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:37:32.819518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.819492 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-kube-api-access-ptjp9" pod "db5fb24d-efda-4edc-aefe-46cef86a2c1b" (UID: "db5fb24d-efda-4edc-aefe-46cef86a2c1b"). InnerVolumeSpecName "kube-api-access-ptjp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:32.819761 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.819736 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-provider-credentials" pod "db5fb24d-efda-4edc-aefe-46cef86a2c1b" (UID: "db5fb24d-efda-4edc-aefe-46cef86a2c1b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:32.919647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.919618 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjp9\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-kube-api-access-ptjp9\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:37:32.919647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:32.919644 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db5fb24d-efda-4edc-aefe-46cef86a2c1b-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:37:33.177884 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.177795 4431 generic.go:356] "Generic (PLEG): container finished" podID="db5fb24d-efda-4edc-aefe-46cef86a2c1b" containerID="0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b" exitCode=0 Oct 30 06:37:33.177884 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.177851 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" Oct 30 06:37:33.178130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.177924 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" event={"ID":"db5fb24d-efda-4edc-aefe-46cef86a2c1b","Type":"ContainerDied","Data":"0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b"} Oct 30 06:37:33.178130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.177985 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d" event={"ID":"db5fb24d-efda-4edc-aefe-46cef86a2c1b","Type":"ContainerDied","Data":"4d355a38c8c5cf8e4df2e2addc533c8f804a8c462941111cb673b07c72a1652f"} Oct 30 06:37:33.178130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.178014 4431 scope.go:122] "RemoveContainer" containerID="0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b" Oct 30 06:37:33.190161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.190147 4431 scope.go:122] "RemoveContainer" containerID="0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b" Oct 30 06:37:33.190542 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:37:33.190523 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b\": container with ID starting with 0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b not found: ID does not exist" containerID="0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b" Oct 30 06:37:33.190625 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.190547 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b"} err="failed to get container status \"0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b\": rpc error: code = NotFound desc = could not find container \"0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b\": container with ID starting with 0b2b682e86e365876af19343f9d4f9313a676578d9ca48b9912fc040dceb208b not found: ID does not exist" Oct 30 06:37:33.230589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.230567 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d"] Oct 30 06:37:33.234511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.234492 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ad1661/payload-processing-external-model-e2e-extmod-a-ad1661-6dd4cnt7d"] Oct 30 06:37:33.338964 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:33.338912 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5fb24d-efda-4edc-aefe-46cef86a2c1b" path="/var/lib/kubelet/pods/db5fb24d-efda-4edc-aefe-46cef86a2c1b/volumes" Oct 30 06:37:33.702825 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:37:33.702789 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:37:45.683931 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:45.683899 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4"] Oct 30 06:37:50.684927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:50.684884 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" podUID="551737c6-162f-4c3b-b2ae-dba44f39e5eb" containerName="payload-processing" containerID="cri-o://cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547" gracePeriod=25 Oct 30 06:37:50.963899 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:50.963871 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:37:51.043013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.042983 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-provider-credentials\") pod \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " Oct 30 06:37:51.043180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.043141 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-plugins-config-volume\") pod \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " Oct 30 06:37:51.043275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.043260 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-kube-api-access-lzlmm\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-kube-api-access-lzlmm\") pod \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " Oct 30 06:37:51.043458 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.043436 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-routing-overlay-volume\") pod \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\" (UID: \"551737c6-162f-4c3b-b2ae-dba44f39e5eb\") " Oct 30 06:37:51.043769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.043616 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-plugins-config-volume" pod "551737c6-162f-4c3b-b2ae-dba44f39e5eb" (UID: "551737c6-162f-4c3b-b2ae-dba44f39e5eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:51.043858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.043797 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-routing-overlay-volume" pod "551737c6-162f-4c3b-b2ae-dba44f39e5eb" (UID: "551737c6-162f-4c3b-b2ae-dba44f39e5eb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:51.044198 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.044176 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:37:51.044336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.044204 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/551737c6-162f-4c3b-b2ae-dba44f39e5eb-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:37:51.045592 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.045567 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-provider-credentials" pod "551737c6-162f-4c3b-b2ae-dba44f39e5eb" (UID: "551737c6-162f-4c3b-b2ae-dba44f39e5eb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:51.045965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.045943 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-kube-api-access-lzlmm" pod "551737c6-162f-4c3b-b2ae-dba44f39e5eb" (UID: "551737c6-162f-4c3b-b2ae-dba44f39e5eb"). InnerVolumeSpecName "kube-api-access-lzlmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:51.146198 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.146161 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:37:51.146198 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.146193 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzlmm\" (UniqueName: \"kubernetes.io/projected/551737c6-162f-4c3b-b2ae-dba44f39e5eb-kube-api-access-lzlmm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:37:51.341932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.341899 4431 generic.go:356] "Generic (PLEG): container finished" podID="551737c6-162f-4c3b-b2ae-dba44f39e5eb" containerID="cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547" exitCode=0 Oct 30 06:37:51.342229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.341944 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" Oct 30 06:37:51.342229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.341974 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" event={"ID":"551737c6-162f-4c3b-b2ae-dba44f39e5eb","Type":"ContainerDied","Data":"cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547"} Oct 30 06:37:51.342229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.342021 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4" event={"ID":"551737c6-162f-4c3b-b2ae-dba44f39e5eb","Type":"ContainerDied","Data":"3a42ff222aa7e12bf3445ed9ba3f12b88e8d596d6657d908a1a173525c446337"} Oct 30 06:37:51.342229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.342047 4431 scope.go:122] "RemoveContainer" containerID="cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547" Oct 30 06:37:51.356137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.356112 4431 scope.go:122] "RemoveContainer" containerID="cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547" Oct 30 06:37:51.356663 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:37:51.356640 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547\": container with ID starting with cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547 not found: ID does not exist" containerID="cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547" Oct 30 06:37:51.356759 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.356673 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547"} err="failed to get container status \"cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547\": rpc error: code = NotFound desc = could not find container \"cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547\": container with ID starting with cefae8d7f1f61a8df35612e553c0f5d7e470d4411c5361d424b94b49e3268547 not found: ID does not exist" Oct 30 06:37:51.393841 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.393814 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4"] Oct 30 06:37:51.398432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:51.398408 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6d1778/payload-processing-external-model-e2e-extmod-b-6d1778-6b9fwhlt4"] Oct 30 06:37:53.339050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:53.339009 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551737c6-162f-4c3b-b2ae-dba44f39e5eb" path="/var/lib/kubelet/pods/551737c6-162f-4c3b-b2ae-dba44f39e5eb/volumes" Oct 30 06:37:54.901364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.901335 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd"] Oct 30 06:37:54.902080 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.902065 4431 cpu_manager.go:414] "containerMap: removing container" podUID="551737c6-162f-4c3b-b2ae-dba44f39e5eb" containerName="payload-processing" Oct 30 06:37:54.902130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.902081 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551737c6-162f-4c3b-b2ae-dba44f39e5eb" containerName="payload-processing" Oct 30 06:37:54.902130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.902113 4431 cpu_manager.go:414] "containerMap: removing container" podUID="db5fb24d-efda-4edc-aefe-46cef86a2c1b" containerName="payload-processing" Oct 30 06:37:54.902130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.902118 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5fb24d-efda-4edc-aefe-46cef86a2c1b" containerName="payload-processing" Oct 30 06:37:54.902232 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.902223 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="551737c6-162f-4c3b-b2ae-dba44f39e5eb" containerName="payload-processing" Oct 30 06:37:54.902266 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.902237 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5fb24d-efda-4edc-aefe-46cef86a2c1b" containerName="payload-processing" Oct 30 06:37:54.907199 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.907179 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:54.935884 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.935860 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2d8df2-openshift-default-dockercfg-nmcfj\"" Oct 30 06:37:54.942506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:54.942067 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd"] Oct 30 06:37:55.011709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.011684 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-envoy\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.011832 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.011719 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-podinfo\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.011832 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.011819 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-data\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.011933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.011915 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzx6f\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-kube-api-access-hzx6f\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.011989 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.011970 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-certs\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.012097 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.012077 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-credential-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.012209 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.012140 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istiod-ca-cert\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.012283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.012217 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.012283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.012253 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-token\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.113981 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istiod-ca-cert\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114030 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114061 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-token\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114140 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-envoy\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114177 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-podinfo\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114348 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-data\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114475 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzx6f\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-kube-api-access-hzx6f\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114545 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114552 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-certs\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114690 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-credential-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114726 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-data\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114857 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-certs\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.114887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114880 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istiod-ca-cert\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.115064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.114993 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-credential-socket\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.116401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.116375 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-envoy\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.116718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.116697 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-podinfo\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.137168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.137149 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-token\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.137353 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.137334 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzx6f\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-kube-api-access-hzx6f\") pod \"e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.222952 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.222931 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:55.560444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.560417 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd"] Oct 30 06:37:55.561547 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:37:55.561523 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2e0884_320a_4cf8_ba2a_55d22ff54988.slice/crio-01661b86dfa6fd01b5918c8bde16e293e52897df14fed8798350492e22f9f1dd WatchSource:0}: Error finding container 01661b86dfa6fd01b5918c8bde16e293e52897df14fed8798350492e22f9f1dd: Status 404 returned error can't find the container with id 01661b86dfa6fd01b5918c8bde16e293e52897df14fed8798350492e22f9f1dd Oct 30 06:37:55.564331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.564271 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:37:55.564411 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.564353 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:37:55.564411 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:55.564386 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:37:56.392732 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:56.392690 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" event={"ID":"6a2e0884-320a-4cf8-ba2a-55d22ff54988","Type":"ContainerStarted","Data":"6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae"} Oct 30 06:37:56.393195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:56.392755 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" event={"ID":"6a2e0884-320a-4cf8-ba2a-55d22ff54988","Type":"ContainerStarted","Data":"01661b86dfa6fd01b5918c8bde16e293e52897df14fed8798350492e22f9f1dd"} Oct 30 06:37:56.452073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:56.452027 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" podStartSLOduration=2.452013129 podStartE2EDuration="2.452013129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:37:56.448325302 +0000 UTC m=+2685.254847051" watchObservedRunningTime="2026-10-30 06:37:56.452013129 +0000 UTC m=+2685.258534877" Oct 30 06:37:57.223664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:57.223629 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:37:57.225082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:57.225058 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:37:57.225200 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:57.225103 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 06:37:58.223959 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:58.223923 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:37:58.224405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:58.223976 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 06:37:59.223447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:59.223412 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:37:59.223656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:37:59.223474 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 06:38:00.223586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:00.223546 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:38:00.224070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:00.223611 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 06:38:01.223906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:01.223874 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:38:01.224360 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:01.223923 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 06:38:02.636050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:02.636005 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:38:02.636856 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:02.636763 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:38:02.638556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:02.638530 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:38:03.702339 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:38:03.702283 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:38:13.400487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.400456 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s"] Oct 30 06:38:13.406082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.406059 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:13.424344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.424291 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s"] Oct 30 06:38:13.431770 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.431733 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c44730\"" Oct 30 06:38:13.431885 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.431814 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c44730-dockercfg-2mcmh\"" Oct 30 06:38:13.469781 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.469748 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb"] Oct 30 06:38:13.474374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.474352 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:13.490534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.490509 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb"] Oct 30 06:38:13.594327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.570752 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d53b413-657e-4353-b14e-75ccd742e78b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb\" (UID: \"2d53b413-657e-4353-b14e-75ccd742e78b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:13.598323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.595507 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d99c\" (UniqueName: \"kubernetes.io/projected/9c4debb0-8735-48b2-a4fd-2bb725083af0-kube-api-access-6d99c\") pod \"payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s\" (UID: \"9c4debb0-8735-48b2-a4fd-2bb725083af0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:13.598323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.595708 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c4debb0-8735-48b2-a4fd-2bb725083af0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s\" (UID: \"9c4debb0-8735-48b2-a4fd-2bb725083af0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:13.598323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.595781 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2trf\" (UniqueName: \"kubernetes.io/projected/2d53b413-657e-4353-b14e-75ccd742e78b-kube-api-access-c2trf\") pod \"payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb\" (UID: \"2d53b413-657e-4353-b14e-75ccd742e78b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:13.724328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.720885 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2trf\" (UniqueName: \"kubernetes.io/projected/2d53b413-657e-4353-b14e-75ccd742e78b-kube-api-access-c2trf\") pod \"payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb\" (UID: \"2d53b413-657e-4353-b14e-75ccd742e78b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:13.724328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.721096 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d53b413-657e-4353-b14e-75ccd742e78b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb\" (UID: \"2d53b413-657e-4353-b14e-75ccd742e78b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:13.724328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.721204 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d99c\" (UniqueName: \"kubernetes.io/projected/9c4debb0-8735-48b2-a4fd-2bb725083af0-kube-api-access-6d99c\") pod \"payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s\" (UID: \"9c4debb0-8735-48b2-a4fd-2bb725083af0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:13.724328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.721333 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c4debb0-8735-48b2-a4fd-2bb725083af0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s\" (UID: \"9c4debb0-8735-48b2-a4fd-2bb725083af0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:13.729377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.729345 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c4debb0-8735-48b2-a4fd-2bb725083af0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s\" (UID: \"9c4debb0-8735-48b2-a4fd-2bb725083af0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:13.767346 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.751867 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d53b413-657e-4353-b14e-75ccd742e78b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb\" (UID: \"2d53b413-657e-4353-b14e-75ccd742e78b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:13.828013 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.827978 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2trf\" (UniqueName: \"kubernetes.io/projected/2d53b413-657e-4353-b14e-75ccd742e78b-kube-api-access-c2trf\") pod \"payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb\" (UID: \"2d53b413-657e-4353-b14e-75ccd742e78b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:13.853728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:13.853698 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d99c\" (UniqueName: \"kubernetes.io/projected/9c4debb0-8735-48b2-a4fd-2bb725083af0-kube-api-access-6d99c\") pod \"payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s\" (UID: \"9c4debb0-8735-48b2-a4fd-2bb725083af0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:14.020503 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.020422 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:14.091412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.091387 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:14.338277 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.338253 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s"] Oct 30 06:38:14.340828 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:38:14.340805 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4debb0_8735_48b2_a4fd_2bb725083af0.slice/crio-9f5663f7a97e2208c512264795fd6baa5dcbbd7a60b4eb5d4c58442469694bfd WatchSource:0}: Error finding container 9f5663f7a97e2208c512264795fd6baa5dcbbd7a60b4eb5d4c58442469694bfd: Status 404 returned error can't find the container with id 9f5663f7a97e2208c512264795fd6baa5dcbbd7a60b4eb5d4c58442469694bfd Oct 30 06:38:14.472333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.472291 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb"] Oct 30 06:38:14.472706 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:38:14.472571 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d53b413_657e_4353_b14e_75ccd742e78b.slice/crio-0fbfefa8fceaac166f72015228257046309fa99abdd6b836b5d94c5dd1267d06 WatchSource:0}: Error finding container 0fbfefa8fceaac166f72015228257046309fa99abdd6b836b5d94c5dd1267d06: Status 404 returned error can't find the container with id 0fbfefa8fceaac166f72015228257046309fa99abdd6b836b5d94c5dd1267d06 Oct 30 06:38:14.855175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.855142 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" event={"ID":"2d53b413-657e-4353-b14e-75ccd742e78b","Type":"ContainerStarted","Data":"b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e"} Oct 30 06:38:14.855381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.855204 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" event={"ID":"2d53b413-657e-4353-b14e-75ccd742e78b","Type":"ContainerStarted","Data":"0fbfefa8fceaac166f72015228257046309fa99abdd6b836b5d94c5dd1267d06"} Oct 30 06:38:14.855381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.855249 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:14.857204 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.857170 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" event={"ID":"9c4debb0-8735-48b2-a4fd-2bb725083af0","Type":"ContainerStarted","Data":"dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b"} Oct 30 06:38:14.857354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.857233 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" event={"ID":"9c4debb0-8735-48b2-a4fd-2bb725083af0","Type":"ContainerStarted","Data":"9f5663f7a97e2208c512264795fd6baa5dcbbd7a60b4eb5d4c58442469694bfd"} Oct 30 06:38:14.857354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.857266 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:14.898710 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.898667 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" podStartSLOduration=1.898654506 podStartE2EDuration="1.898654506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:14.895850097 +0000 UTC m=+2703.702371844" watchObservedRunningTime="2026-10-30 06:38:14.898654506 +0000 UTC m=+2703.705176253" Oct 30 06:38:14.942089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:14.942041 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" podStartSLOduration=1.9420269220000002 podStartE2EDuration="1.942026922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:14.937223904 +0000 UTC m=+2703.743745652" watchObservedRunningTime="2026-10-30 06:38:14.942026922 +0000 UTC m=+2703.748548761" Oct 30 06:38:25.868224 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:25.868146 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:38:25.868224 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:25.868222 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:38:33.702885 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:38:33.702847 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:38:37.793999 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.793963 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq"] Oct 30 06:38:37.800073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.800042 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:37.828612 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.828580 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq"] Oct 30 06:38:37.833923 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.833896 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"agc-cred-a726bda01017\"" Oct 30 06:38:37.834097 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.833933 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"agc-cred-4b4eb8b1d03a\"" Oct 30 06:38:37.834249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.834044 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"openshift-service-ca.crt\"" Oct 30 06:38:37.834249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.834052 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"kube-root-ca.crt\"" Oct 30 06:38:37.834249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.834156 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b8ca49\"" Oct 30 06:38:37.834531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.834175 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"routing-overlay\"" Oct 30 06:38:37.834531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.834347 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b8ca49\"/\"payload-processing-external-model-e2e--7e273d00-dockercfg-t26z8\"" Oct 30 06:38:37.942155 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.942120 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:37.942384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.942253 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:37.942466 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.942392 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:37.942529 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:37.942515 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r4r5\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-kube-api-access-4r4r5\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.043772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.043742 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.043915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.043819 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.043915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.043901 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.043994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.043977 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r4r5\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-kube-api-access-4r4r5\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.044690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.044627 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.044690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.044635 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.046317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.046281 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.067227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.067208 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r4r5\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-kube-api-access-4r4r5\") pod \"payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.129814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.129790 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:38.463882 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:38.463852 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq"] Oct 30 06:38:38.464708 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:38:38.464685 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad444b0_05db_4a98_83c7_54372ae0caa7.slice/crio-59c7959f85de5c7f796de8718eea8345e763cadbe7b6afd6df14655475ea3e96 WatchSource:0}: Error finding container 59c7959f85de5c7f796de8718eea8345e763cadbe7b6afd6df14655475ea3e96: Status 404 returned error can't find the container with id 59c7959f85de5c7f796de8718eea8345e763cadbe7b6afd6df14655475ea3e96 Oct 30 06:38:39.258622 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:39.258584 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" event={"ID":"7ad444b0-05db-4a98-83c7-54372ae0caa7","Type":"ContainerStarted","Data":"b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8"} Oct 30 06:38:39.259066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:39.258664 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" event={"ID":"7ad444b0-05db-4a98-83c7-54372ae0caa7","Type":"ContainerStarted","Data":"59c7959f85de5c7f796de8718eea8345e763cadbe7b6afd6df14655475ea3e96"} Oct 30 06:38:39.259066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:39.258731 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:39.300654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:39.300605 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" podStartSLOduration=2.300593179 podStartE2EDuration="2.300593179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:39.29923394 +0000 UTC m=+2728.105755687" watchObservedRunningTime="2026-10-30 06:38:39.300593179 +0000 UTC m=+2728.107114927" Oct 30 06:38:44.035901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.035867 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7"] Oct 30 06:38:44.040378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.040359 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.058275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.058251 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7"] Oct 30 06:38:44.065450 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.065422 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"kube-root-ca.crt\"" Oct 30 06:38:44.065450 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.065421 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"agc-cred-1f70c745963f\"" Oct 30 06:38:44.065603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.065431 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"agc-cred-c32bb415fa0e\"" Oct 30 06:38:44.065603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.065510 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"payload-processing-external-model-e2e--d83828a5-dockercfg-cqlg5\"" Oct 30 06:38:44.065750 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.065727 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"openshift-service-ca.crt\"" Oct 30 06:38:44.068685 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.068664 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2d2923\"" Oct 30 06:38:44.068787 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.068720 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d2923\"/\"routing-overlay\"" Oct 30 06:38:44.138612 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.138585 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q9bl\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-kube-api-access-6q9bl\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.138742 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.138658 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.138742 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.138705 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.138814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.138762 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.240161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.240115 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q9bl\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-kube-api-access-6q9bl\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.240161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.240163 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.240367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.240211 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.240367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.240255 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.241110 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.241082 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.241225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.241149 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.242877 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.242854 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.263677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.263652 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q9bl\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-kube-api-access-6q9bl\") pod \"payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.389995 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.389906 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:44.715527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:44.715504 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7"] Oct 30 06:38:44.716107 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:38:44.716086 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9694460e_38ab_48bb_9f72_824a44c171fe.slice/crio-e1d1b8d87747037874b27359d6fbeb85097b6bab0e81db60034e2971962b3f75 WatchSource:0}: Error finding container e1d1b8d87747037874b27359d6fbeb85097b6bab0e81db60034e2971962b3f75: Status 404 returned error can't find the container with id e1d1b8d87747037874b27359d6fbeb85097b6bab0e81db60034e2971962b3f75 Oct 30 06:38:45.412533 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:45.412496 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" event={"ID":"9694460e-38ab-48bb-9f72-824a44c171fe","Type":"ContainerStarted","Data":"a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44"} Oct 30 06:38:45.412934 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:45.412559 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:38:45.412934 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:45.412581 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" event={"ID":"9694460e-38ab-48bb-9f72-824a44c171fe","Type":"ContainerStarted","Data":"e1d1b8d87747037874b27359d6fbeb85097b6bab0e81db60034e2971962b3f75"} Oct 30 06:38:45.458145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:45.458091 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" podStartSLOduration=1.458075845 podStartE2EDuration="1.458075845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:45.45383488 +0000 UTC m=+2734.260356638" watchObservedRunningTime="2026-10-30 06:38:45.458075845 +0000 UTC m=+2734.264597593" Oct 30 06:38:50.268194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:50.268161 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:38:56.080118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:56.080045 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9"] Oct 30 06:38:56.080586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:56.080287 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" containerID="cri-o://4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4" gracePeriod=30 Oct 30 06:38:56.092387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:56.092357 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg"] Oct 30 06:38:56.092624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:56.092599 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" podUID="728404ea-ba21-420d-bc69-170c41334445" containerName="istio-proxy" containerID="cri-o://c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4" gracePeriod=30 Oct 30 06:38:56.096561 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:56.096537 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl"] Oct 30 06:38:56.096759 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:56.096739 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" podUID="207c013b-e701-44ae-8363-df664833ec83" containerName="istio-proxy" containerID="cri-o://fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242" gracePeriod=30 Oct 30 06:38:56.424622 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:38:56.424527 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:39:01.110997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.110967 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd"] Oct 30 06:39:01.116569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.116549 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.134180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.134151 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd"] Oct 30 06:39:01.175271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.175233 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.175451 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.175295 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kblq2\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-kube-api-access-kblq2\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.175521 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.175494 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.175644 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.175617 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.175711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.175665 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.175816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.175792 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.175900 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.175870 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.175971 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.175903 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.175971 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.175936 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.280840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.280507 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.280840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.280612 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.280840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.280648 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.280840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.280680 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.280840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.280798 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.280840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.280836 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kblq2\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-kube-api-access-kblq2\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.281251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.280982 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.281251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.281063 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.281251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.281111 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.281452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.281383 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.281603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.281536 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.281603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.281575 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.282596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.282227 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.282596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.282398 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.285473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.285250 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.288237 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.288208 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.308348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.307908 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kblq2\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-kube-api-access-kblq2\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.308693 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.308612 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7t4cd\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.376464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.376432 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:39:01.381780 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.381748 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-socket\") pod \"728404ea-ba21-420d-bc69-170c41334445\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " Oct 30 06:39:01.381942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.381847 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-data\") pod \"728404ea-ba21-420d-bc69-170c41334445\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " Oct 30 06:39:01.381942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.381876 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-certs\") pod \"728404ea-ba21-420d-bc69-170c41334445\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " Oct 30 06:39:01.382073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.381936 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728404ea-ba21-420d-bc69-170c41334445-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728404ea-ba21-420d-bc69-170c41334445-istiod-ca-cert\") pod \"728404ea-ba21-420d-bc69-170c41334445\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " Oct 30 06:39:01.382073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.382049 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-credential-socket\") pod \"728404ea-ba21-420d-bc69-170c41334445\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " Oct 30 06:39:01.382186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.382067 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-socket" pod "728404ea-ba21-420d-bc69-170c41334445" (UID: "728404ea-ba21-420d-bc69-170c41334445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.382186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.382100 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-kube-api-access-kq7fq\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-kube-api-access-kq7fq\") pod \"728404ea-ba21-420d-bc69-170c41334445\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " Oct 30 06:39:01.382186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.382130 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-envoy\") pod \"728404ea-ba21-420d-bc69-170c41334445\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " Oct 30 06:39:01.382186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.382158 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-istio-token\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-istio-token\") pod \"728404ea-ba21-420d-bc69-170c41334445\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " Oct 30 06:39:01.382422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.382342 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-certs" pod "728404ea-ba21-420d-bc69-170c41334445" (UID: "728404ea-ba21-420d-bc69-170c41334445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.382422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.382397 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/728404ea-ba21-420d-bc69-170c41334445-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728404ea-ba21-420d-bc69-170c41334445-istio-podinfo\") pod \"728404ea-ba21-420d-bc69-170c41334445\" (UID: \"728404ea-ba21-420d-bc69-170c41334445\") " Oct 30 06:39:01.382553 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.382533 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-data" pod "728404ea-ba21-420d-bc69-170c41334445" (UID: "728404ea-ba21-420d-bc69-170c41334445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.383194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.383164 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.383328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.383196 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.383328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.383210 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.385214 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.385188 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-kube-api-access-kq7fq" pod "728404ea-ba21-420d-bc69-170c41334445" (UID: "728404ea-ba21-420d-bc69-170c41334445"). InnerVolumeSpecName "kube-api-access-kq7fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:01.385761 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.385584 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-envoy" pod "728404ea-ba21-420d-bc69-170c41334445" (UID: "728404ea-ba21-420d-bc69-170c41334445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.385761 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.385649 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728404ea-ba21-420d-bc69-170c41334445-istiod-ca-cert" pod "728404ea-ba21-420d-bc69-170c41334445" (UID: "728404ea-ba21-420d-bc69-170c41334445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:01.385982 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.385956 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-credential-socket" pod "728404ea-ba21-420d-bc69-170c41334445" (UID: "728404ea-ba21-420d-bc69-170c41334445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.388260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.388216 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/728404ea-ba21-420d-bc69-170c41334445-istio-podinfo" pod "728404ea-ba21-420d-bc69-170c41334445" (UID: "728404ea-ba21-420d-bc69-170c41334445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:39:01.388812 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.388774 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-istio-token" pod "728404ea-ba21-420d-bc69-170c41334445" (UID: "728404ea-ba21-420d-bc69-170c41334445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:01.459175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.459146 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:39:01.463909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.463888 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:39:01.483747 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.483725 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-certs\") pod \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " Oct 30 06:39:01.483867 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.483785 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-data\") pod \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " Oct 30 06:39:01.483867 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.483808 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-kube-api-access-wjx2m\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-kube-api-access-wjx2m\") pod \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " Oct 30 06:39:01.483867 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.483851 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-podinfo\") pod \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " Oct 30 06:39:01.484039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.483878 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-credential-socket\") pod \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " Oct 30 06:39:01.484095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484058 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-certs" pod "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" (UID: "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.484148 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484077 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-data" pod "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" (UID: "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.484364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484346 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/207c013b-e701-44ae-8363-df664833ec83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207c013b-e701-44ae-8363-df664833ec83-istio-podinfo\") pod \"207c013b-e701-44ae-8363-df664833ec83\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " Oct 30 06:39:01.484468 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484363 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-credential-socket" pod "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" (UID: "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.484468 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484384 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-envoy\") pod \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " Oct 30 06:39:01.484468 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484421 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istiod-ca-cert\") pod \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " Oct 30 06:39:01.484633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484478 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-data\") pod \"207c013b-e701-44ae-8363-df664833ec83\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " Oct 30 06:39:01.484633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484520 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-credential-socket\") pod \"207c013b-e701-44ae-8363-df664833ec83\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " Oct 30 06:39:01.484633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484572 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-kube-api-access-mrh44\") pod \"207c013b-e701-44ae-8363-df664833ec83\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " Oct 30 06:39:01.484633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484607 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-certs\") pod \"207c013b-e701-44ae-8363-df664833ec83\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " Oct 30 06:39:01.484843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484642 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-socket\") pod \"207c013b-e701-44ae-8363-df664833ec83\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " Oct 30 06:39:01.484843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484704 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-envoy\") pod \"207c013b-e701-44ae-8363-df664833ec83\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " Oct 30 06:39:01.484843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484732 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207c013b-e701-44ae-8363-df664833ec83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207c013b-e701-44ae-8363-df664833ec83-istiod-ca-cert\") pod \"207c013b-e701-44ae-8363-df664833ec83\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " Oct 30 06:39:01.484843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484782 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-token\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-token\") pod \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " Oct 30 06:39:01.485060 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484872 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istiod-ca-cert" pod "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" (UID: "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:01.485060 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484897 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-istio-token\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-istio-token\") pod \"207c013b-e701-44ae-8363-df664833ec83\" (UID: \"207c013b-e701-44ae-8363-df664833ec83\") " Oct 30 06:39:01.485060 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.484938 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-socket\") pod \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\" (UID: \"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd\") " Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.485500 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-socket" pod "207c013b-e701-44ae-8363-df664833ec83" (UID: "207c013b-e701-44ae-8363-df664833ec83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.485708 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-socket" pod "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" (UID: "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.485904 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-data" pod "207c013b-e701-44ae-8363-df664833ec83" (UID: "207c013b-e701-44ae-8363-df664833ec83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.485953 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.485974 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.485988 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486003 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728404ea-ba21-420d-bc69-170c41334445-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486018 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486032 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728404ea-ba21-420d-bc69-170c41334445-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486047 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486061 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486076 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7fq\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-kube-api-access-kq7fq\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486076 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-credential-socket" pod "207c013b-e701-44ae-8363-df664833ec83" (UID: "207c013b-e701-44ae-8363-df664833ec83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486089 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728404ea-ba21-420d-bc69-170c41334445-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.486238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486102 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728404ea-ba21-420d-bc69-170c41334445-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.487093 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486779 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-kube-api-access-wjx2m" pod "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" (UID: "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd"). InnerVolumeSpecName "kube-api-access-wjx2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:01.487093 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.486983 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/207c013b-e701-44ae-8363-df664833ec83-istio-podinfo" pod "207c013b-e701-44ae-8363-df664833ec83" (UID: "207c013b-e701-44ae-8363-df664833ec83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:39:01.487249 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.487224 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-envoy" pod "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" (UID: "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.487905 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.487617 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-certs" pod "207c013b-e701-44ae-8363-df664833ec83" (UID: "207c013b-e701-44ae-8363-df664833ec83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.487905 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.487876 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207c013b-e701-44ae-8363-df664833ec83-istiod-ca-cert" pod "207c013b-e701-44ae-8363-df664833ec83" (UID: "207c013b-e701-44ae-8363-df664833ec83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:01.488519 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.488481 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-podinfo" pod "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" (UID: "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:39:01.489723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.489692 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-kube-api-access-mrh44" pod "207c013b-e701-44ae-8363-df664833ec83" (UID: "207c013b-e701-44ae-8363-df664833ec83"). InnerVolumeSpecName "kube-api-access-mrh44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:01.490084 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.490063 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-token" pod "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" (UID: "7cdf6871-2729-4b45-9b72-4b2bcd16d1bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:01.490204 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.490186 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-istio-token" pod "207c013b-e701-44ae-8363-df664833ec83" (UID: "207c013b-e701-44ae-8363-df664833ec83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:01.490892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.490873 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-envoy" pod "207c013b-e701-44ae-8363-df664833ec83" (UID: "207c013b-e701-44ae-8363-df664833ec83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:01.559679 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.559647 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:01.562758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.562726 4431 generic.go:356] "Generic (PLEG): container finished" podID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerID="4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4" exitCode=0 Oct 30 06:39:01.562894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.562796 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" Oct 30 06:39:01.562894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.562822 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" event={"ID":"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd","Type":"ContainerDied","Data":"4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4"} Oct 30 06:39:01.562894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.562880 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9" event={"ID":"7cdf6871-2729-4b45-9b72-4b2bcd16d1bd","Type":"ContainerDied","Data":"d6a8dd8d2682370e5157d883c4aab0c1e108ab6d3c7886fda929d4c082827b2b"} Oct 30 06:39:01.563076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.562911 4431 scope.go:122] "RemoveContainer" containerID="4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4" Oct 30 06:39:01.566576 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.566550 4431 generic.go:356] "Generic (PLEG): container finished" podID="728404ea-ba21-420d-bc69-170c41334445" containerID="c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4" exitCode=0 Oct 30 06:39:01.566861 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.566837 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" event={"ID":"728404ea-ba21-420d-bc69-170c41334445","Type":"ContainerDied","Data":"c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4"} Oct 30 06:39:01.567028 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.567009 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" event={"ID":"728404ea-ba21-420d-bc69-170c41334445","Type":"ContainerDied","Data":"e714470a65cef45113e53ddef1ba30cf6a6566e41e32afa163f64a3fe8d3ab2f"} Oct 30 06:39:01.567212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.567195 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg" Oct 30 06:39:01.578319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.574752 4431 generic.go:356] "Generic (PLEG): container finished" podID="207c013b-e701-44ae-8363-df664833ec83" containerID="fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242" exitCode=0 Oct 30 06:39:01.578319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.574791 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" event={"ID":"207c013b-e701-44ae-8363-df664833ec83","Type":"ContainerDied","Data":"fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242"} Oct 30 06:39:01.578319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.574829 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" event={"ID":"207c013b-e701-44ae-8363-df664833ec83","Type":"ContainerDied","Data":"7b9b71db46381d89c5c717c7a4da48ac9213b9c6a90e6c53c262a21b331b6a1d"} Oct 30 06:39:01.578319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.574900 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl" Oct 30 06:39:01.587278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587257 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587280 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587332 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrh44\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-kube-api-access-mrh44\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587343 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587352 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207c013b-e701-44ae-8363-df664833ec83-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587361 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207c013b-e701-44ae-8363-df664833ec83-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587370 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587377 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207c013b-e701-44ae-8363-df664833ec83-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587389 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjx2m\" (UniqueName: \"kubernetes.io/projected/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-kube-api-access-wjx2m\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587400 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587414 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207c013b-e701-44ae-8363-df664833ec83-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587428 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587442 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:01.587880 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587401 4431 scope.go:122] "RemoveContainer" containerID="4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4" Oct 30 06:39:01.587880 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:39:01.587753 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4\": container with ID starting with 4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4 not found: ID does not exist" containerID="4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4" Oct 30 06:39:01.587880 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587788 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4"} err="failed to get container status \"4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4\": rpc error: code = NotFound desc = could not find container \"4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4\": container with ID starting with 4ac3f240cc4b1903719b81b3e9a1b473f955a38396fffa31d2f0a40351a5d5f4 not found: ID does not exist" Oct 30 06:39:01.587880 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.587809 4431 scope.go:122] "RemoveContainer" containerID="c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4" Oct 30 06:39:01.602928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.602864 4431 scope.go:122] "RemoveContainer" containerID="c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4" Oct 30 06:39:01.603330 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:39:01.603165 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4\": container with ID starting with c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4 not found: ID does not exist" containerID="c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4" Oct 30 06:39:01.603330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.603202 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4"} err="failed to get container status \"c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4\": rpc error: code = NotFound desc = could not find container \"c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4\": container with ID starting with c6ff9a60f4b482d5b725b31584fbf1d1d7c92dd8247cc85c87e2d148b705e9d4 not found: ID does not exist" Oct 30 06:39:01.603330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.603227 4431 scope.go:122] "RemoveContainer" containerID="fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242" Oct 30 06:39:01.618502 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.618479 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9"] Oct 30 06:39:01.619467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.619446 4431 scope.go:122] "RemoveContainer" containerID="fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242" Oct 30 06:39:01.620106 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:39:01.620071 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242\": container with ID starting with fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242 not found: ID does not exist" containerID="fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242" Oct 30 06:39:01.620165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.620114 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242"} err="failed to get container status \"fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242\": rpc error: code = NotFound desc = could not find container \"fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242\": container with ID starting with fa1fcaf81f6acf0871a5bc9339fc5f5ef89fd11fe52a84e4f717d424c27b5242 not found: ID does not exist" Oct 30 06:39:01.624676 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.624656 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-b4vn9"] Oct 30 06:39:01.703578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.703535 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg"] Oct 30 06:39:01.710779 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.710751 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-j4hgg"] Oct 30 06:39:01.748334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.748274 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl"] Oct 30 06:39:01.753579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.753554 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-fkvcl"] Oct 30 06:39:01.908147 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.908116 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd"] Oct 30 06:39:01.908571 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:39:01.908542 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2b3564_82c5_4f7a_8b4e_9bdec23c5632.slice/crio-1838c877dca72d1ed7dd8d2cd4b7093f2e92720ccffc14efa84564aa19d62cca WatchSource:0}: Error finding container 1838c877dca72d1ed7dd8d2cd4b7093f2e92720ccffc14efa84564aa19d62cca: Status 404 returned error can't find the container with id 1838c877dca72d1ed7dd8d2cd4b7093f2e92720ccffc14efa84564aa19d62cca Oct 30 06:39:01.911665 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.911620 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:39:01.911788 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.911693 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:39:01.911788 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:01.911722 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:39:02.594345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:02.594286 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" event={"ID":"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632","Type":"ContainerStarted","Data":"491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9"} Oct 30 06:39:02.595238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:02.595211 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" event={"ID":"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632","Type":"ContainerStarted","Data":"1838c877dca72d1ed7dd8d2cd4b7093f2e92720ccffc14efa84564aa19d62cca"} Oct 30 06:39:02.687327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:02.686655 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podStartSLOduration=1.686630685 podStartE2EDuration="1.686630685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:02.677807884 +0000 UTC m=+2751.484329685" watchObservedRunningTime="2026-10-30 06:39:02.686630685 +0000 UTC m=+2751.493152437" Oct 30 06:39:03.340587 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:03.340556 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207c013b-e701-44ae-8363-df664833ec83" path="/var/lib/kubelet/pods/207c013b-e701-44ae-8363-df664833ec83/volumes" Oct 30 06:39:03.341151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:03.341137 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728404ea-ba21-420d-bc69-170c41334445" path="/var/lib/kubelet/pods/728404ea-ba21-420d-bc69-170c41334445/volumes" Oct 30 06:39:03.341728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:03.341710 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" path="/var/lib/kubelet/pods/7cdf6871-2729-4b45-9b72-4b2bcd16d1bd/volumes" Oct 30 06:39:03.575322 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:03.569634 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:03.584327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:03.581440 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7t4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:39:03.584327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:03.581502 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 06:39:03.711185 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:39:03.711150 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:39:04.572337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:04.571686 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7t4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:39:04.572337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:04.571750 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 06:39:05.565335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:05.564670 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7t4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:39:05.565335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:05.564770 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 06:39:06.569324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:06.566641 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7t4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:39:06.569324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:06.566740 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 06:39:07.579326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:07.568622 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7t4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:39:07.579326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:07.568678 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 06:39:08.560422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:08.560384 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7t4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:39:08.560611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:08.560456 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 06:39:09.560566 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:09.560530 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7t4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:39:09.561040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:09.560596 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 06:39:10.642201 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:10.641747 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:10.643951 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:10.643928 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:10.646159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:10.646139 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:39:17.871092 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.871058 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9"] Oct 30 06:39:17.871941 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.871918 4431 cpu_manager.go:414] "containerMap: removing container" podUID="207c013b-e701-44ae-8363-df664833ec83" containerName="istio-proxy" Oct 30 06:39:17.872050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.871943 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207c013b-e701-44ae-8363-df664833ec83" containerName="istio-proxy" Oct 30 06:39:17.872050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.871962 4431 cpu_manager.go:414] "containerMap: removing container" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" Oct 30 06:39:17.872050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.871969 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" Oct 30 06:39:17.872050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.872012 4431 cpu_manager.go:414] "containerMap: removing container" podUID="728404ea-ba21-420d-bc69-170c41334445" containerName="istio-proxy" Oct 30 06:39:17.872050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.872020 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728404ea-ba21-420d-bc69-170c41334445" containerName="istio-proxy" Oct 30 06:39:17.872296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.872212 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="207c013b-e701-44ae-8363-df664833ec83" containerName="istio-proxy" Oct 30 06:39:17.872296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.872227 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdf6871-2729-4b45-9b72-4b2bcd16d1bd" containerName="istio-proxy" Oct 30 06:39:17.872296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.872243 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="728404ea-ba21-420d-bc69-170c41334445" containerName="istio-proxy" Oct 30 06:39:17.878120 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.878092 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:17.916810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.916722 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"agc-cred-c41c628c8b52\"" Oct 30 06:39:17.917459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.917422 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"routing-overlay\"" Oct 30 06:39:17.917941 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.917751 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"openshift-service-ca.crt\"" Oct 30 06:39:17.918061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.917803 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"kube-root-ca.crt\"" Oct 30 06:39:17.918132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.917880 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d8df2\"" Oct 30 06:39:17.944236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.944205 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9"] Oct 30 06:39:17.997778 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.996989 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"agc-cred-9523b214660a\"" Oct 30 06:39:17.997778 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:17.997127 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"payload-processing-external-model-e2e--17d17c8c-dockercfg-7b6bq\"" Oct 30 06:39:18.005974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.005942 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.006169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.005981 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.006169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.006119 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpgpc\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-kube-api-access-bpgpc\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.006331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.006175 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.145506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.140504 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.157692 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.157617 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.158057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.158033 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpgpc\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-kube-api-access-bpgpc\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.186414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.186370 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.210461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.187718 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.210714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.153110 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.281261 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.281228 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpgpc\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-kube-api-access-bpgpc\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.284612 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.284521 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.563850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.563813 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:18.870344 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:39:18.870311 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62be59ea_4ef4_4a67_b521_16cba75dd82c.slice/crio-5c0fc90d54f4e7f5833fe51fc943a160982f1940dc1459d17a4a4f4b5e2825b4 WatchSource:0}: Error finding container 5c0fc90d54f4e7f5833fe51fc943a160982f1940dc1459d17a4a4f4b5e2825b4: Status 404 returned error can't find the container with id 5c0fc90d54f4e7f5833fe51fc943a160982f1940dc1459d17a4a4f4b5e2825b4 Oct 30 06:39:18.871312 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:18.871277 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9"] Oct 30 06:39:19.722273 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:19.722238 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" event={"ID":"62be59ea-4ef4-4a67-b521-16cba75dd82c","Type":"ContainerStarted","Data":"481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9"} Oct 30 06:39:19.722500 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:19.722296 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" event={"ID":"62be59ea-4ef4-4a67-b521-16cba75dd82c","Type":"ContainerStarted","Data":"5c0fc90d54f4e7f5833fe51fc943a160982f1940dc1459d17a4a4f4b5e2825b4"} Oct 30 06:39:19.722500 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:19.722395 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:19.796808 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:19.796742 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" podStartSLOduration=2.796721612 podStartE2EDuration="2.796721612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:19.787963654 +0000 UTC m=+2768.594485404" watchObservedRunningTime="2026-10-30 06:39:19.796721612 +0000 UTC m=+2768.603243372" Oct 30 06:39:29.913332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:29.913288 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9"] Oct 30 06:39:29.916417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:29.916391 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:29.946337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:29.946289 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6"] Oct 30 06:39:29.951067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:29.951046 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:29.971212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:29.971186 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6"] Oct 30 06:39:30.097338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.097290 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.097570 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.097542 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.097634 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.097593 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.097880 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.097851 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxx4r\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-kube-api-access-lxx4r\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.200320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.199436 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxx4r\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-kube-api-access-lxx4r\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.200320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.199612 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.200320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.199779 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.200320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.199812 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.208320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.200882 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.208320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.202269 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.247322 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.241104 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.339098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.333383 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxx4r\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-kube-api-access-lxx4r\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.585381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.584612 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:30.939877 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:30.939849 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6"] Oct 30 06:39:30.940894 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:39:30.940868 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef5f952_d068_4817_9f8d_d76642d13d40.slice/crio-cc5dbc704fee5d0f145de2d39ab409df45358447f8a47ce089474cf272be10ef WatchSource:0}: Error finding container cc5dbc704fee5d0f145de2d39ab409df45358447f8a47ce089474cf272be10ef: Status 404 returned error can't find the container with id cc5dbc704fee5d0f145de2d39ab409df45358447f8a47ce089474cf272be10ef Oct 30 06:39:31.239065 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.239033 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg"] Oct 30 06:39:31.243671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.243650 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.265997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.265973 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.266118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.266018 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.266182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.266165 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.266247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.266211 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edcace5-3093-479b-9b0f-3d89251b214a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.266324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.266242 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.266442 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.266423 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbtmd\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-kube-api-access-zbtmd\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.266502 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.266461 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.266603 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.266588 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.266660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.266614 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edcace5-3093-479b-9b0f-3d89251b214a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.276983 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.276957 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg"] Oct 30 06:39:31.368533 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.368502 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.368717 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.368573 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.368717 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.368606 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edcace5-3093-479b-9b0f-3d89251b214a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.368717 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.368625 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.368911 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.368728 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbtmd\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-kube-api-access-zbtmd\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.368911 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.368796 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.368911 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.368886 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.369070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.368918 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edcace5-3093-479b-9b0f-3d89251b214a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.369070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.369058 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.369182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.369159 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.369274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.369250 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.369365 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.369334 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.369519 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.369500 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.369737 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.369714 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edcace5-3093-479b-9b0f-3d89251b214a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.371116 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.371087 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.371375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.371357 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edcace5-3093-479b-9b0f-3d89251b214a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.391241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.391221 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.391490 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.391469 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbtmd\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-kube-api-access-zbtmd\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4gvmg\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.558086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.558019 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:31.805700 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.805666 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" event={"ID":"3ef5f952-d068-4817-9f8d-d76642d13d40","Type":"ContainerStarted","Data":"4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50"} Oct 30 06:39:31.805889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.805720 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:31.805889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.805737 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" event={"ID":"3ef5f952-d068-4817-9f8d-d76642d13d40","Type":"ContainerStarted","Data":"cc5dbc704fee5d0f145de2d39ab409df45358447f8a47ce089474cf272be10ef"} Oct 30 06:39:31.847845 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.847750 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" podStartSLOduration=2.847731273 podStartE2EDuration="2.847731273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:31.845570245 +0000 UTC m=+2780.652091994" watchObservedRunningTime="2026-10-30 06:39:31.847731273 +0000 UTC m=+2780.654253026" Oct 30 06:39:31.892610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.892586 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg"] Oct 30 06:39:31.893043 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:39:31.893023 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edcace5_3093_479b_9b0f_3d89251b214a.slice/crio-b6d229f255e7753a2f845138e1a10d2b4a26b3756f2436c0899827a55e0ca527 WatchSource:0}: Error finding container b6d229f255e7753a2f845138e1a10d2b4a26b3756f2436c0899827a55e0ca527: Status 404 returned error can't find the container with id b6d229f255e7753a2f845138e1a10d2b4a26b3756f2436c0899827a55e0ca527 Oct 30 06:39:31.895639 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.895611 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:39:31.895721 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.895672 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:39:31.895721 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:31.895702 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:39:32.814806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:32.814770 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" event={"ID":"8edcace5-3093-479b-9b0f-3d89251b214a","Type":"ContainerStarted","Data":"38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb"} Oct 30 06:39:32.815326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:32.814828 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" event={"ID":"8edcace5-3093-479b-9b0f-3d89251b214a","Type":"ContainerStarted","Data":"b6d229f255e7753a2f845138e1a10d2b4a26b3756f2436c0899827a55e0ca527"} Oct 30 06:39:32.860206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:32.860159 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podStartSLOduration=1.8601453110000001 podStartE2EDuration="1.860145311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:32.859952413 +0000 UTC m=+2781.666474161" watchObservedRunningTime="2026-10-30 06:39:32.860145311 +0000 UTC m=+2781.666667059" Oct 30 06:39:33.558862 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:33.558830 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:33.588326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:33.573449 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4gvmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:39:33.588326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:33.573515 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 06:39:33.713085 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:39:33.713038 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:39:34.558593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:34.558561 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4gvmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:39:34.559023 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:34.558617 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 06:39:34.927388 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:34.924768 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" podUID="62be59ea-4ef4-4a67-b521-16cba75dd82c" containerName="payload-processing" containerID="cri-o://481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9" gracePeriod=25 Oct 30 06:39:35.338317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.338283 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:35.490435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.490349 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-routing-overlay-volume\") pod \"62be59ea-4ef4-4a67-b521-16cba75dd82c\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " Oct 30 06:39:35.490435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.490417 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-kube-api-access-bpgpc\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-kube-api-access-bpgpc\") pod \"62be59ea-4ef4-4a67-b521-16cba75dd82c\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " Oct 30 06:39:35.490669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.490484 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-plugins-config-volume\") pod \"62be59ea-4ef4-4a67-b521-16cba75dd82c\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " Oct 30 06:39:35.490669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.490561 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-provider-credentials\") pod \"62be59ea-4ef4-4a67-b521-16cba75dd82c\" (UID: \"62be59ea-4ef4-4a67-b521-16cba75dd82c\") " Oct 30 06:39:35.490797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.490715 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-routing-overlay-volume" pod "62be59ea-4ef4-4a67-b521-16cba75dd82c" (UID: "62be59ea-4ef4-4a67-b521-16cba75dd82c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:35.490963 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.490936 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-plugins-config-volume" pod "62be59ea-4ef4-4a67-b521-16cba75dd82c" (UID: "62be59ea-4ef4-4a67-b521-16cba75dd82c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:35.491620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.491596 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:35.494284 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.493898 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62be59ea-4ef4-4a67-b521-16cba75dd82c-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:35.494284 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.493528 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-provider-credentials" pod "62be59ea-4ef4-4a67-b521-16cba75dd82c" (UID: "62be59ea-4ef4-4a67-b521-16cba75dd82c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:35.494284 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.494266 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-kube-api-access-bpgpc" pod "62be59ea-4ef4-4a67-b521-16cba75dd82c" (UID: "62be59ea-4ef4-4a67-b521-16cba75dd82c"). InnerVolumeSpecName "kube-api-access-bpgpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:35.559083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.559052 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4gvmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:39:35.559498 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.559113 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 06:39:35.595593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.595557 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpgpc\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-kube-api-access-bpgpc\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:35.595593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.595593 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62be59ea-4ef4-4a67-b521-16cba75dd82c-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:35.885380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.880491 4431 generic.go:356] "Generic (PLEG): container finished" podID="62be59ea-4ef4-4a67-b521-16cba75dd82c" containerID="481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9" exitCode=0 Oct 30 06:39:35.885380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.880701 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" event={"ID":"62be59ea-4ef4-4a67-b521-16cba75dd82c","Type":"ContainerDied","Data":"481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9"} Oct 30 06:39:35.885380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.880758 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" event={"ID":"62be59ea-4ef4-4a67-b521-16cba75dd82c","Type":"ContainerDied","Data":"5c0fc90d54f4e7f5833fe51fc943a160982f1940dc1459d17a4a4f4b5e2825b4"} Oct 30 06:39:35.885380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.880789 4431 scope.go:122] "RemoveContainer" containerID="481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9" Oct 30 06:39:35.885380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:35.880964 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9" Oct 30 06:39:36.100704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:36.100629 4431 scope.go:122] "RemoveContainer" containerID="481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9" Oct 30 06:39:36.101554 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:39:36.101521 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9\": container with ID starting with 481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9 not found: ID does not exist" containerID="481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9" Oct 30 06:39:36.101691 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:36.101566 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9"} err="failed to get container status \"481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9\": rpc error: code = NotFound desc = could not find container \"481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9\": container with ID starting with 481a0a140c0b8f43673187ef175478aa7a460942b73464416c9d30e3a92c6df9 not found: ID does not exist" Oct 30 06:39:36.116437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:36.116395 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9"] Oct 30 06:39:36.146340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:36.146242 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-54f7cbzl9"] Oct 30 06:39:36.558729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:36.558689 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4gvmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:39:36.558912 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:36.558770 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 06:39:37.339712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:37.339681 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62be59ea-4ef4-4a67-b521-16cba75dd82c" path="/var/lib/kubelet/pods/62be59ea-4ef4-4a67-b521-16cba75dd82c/volumes" Oct 30 06:39:37.558897 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:37.558856 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4gvmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:39:37.559072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:37.558933 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 06:39:38.558690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:38.558655 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4gvmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:39:38.559088 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:38.558707 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 06:39:39.223089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:39.223049 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2"] Oct 30 06:39:39.223393 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:39.223355 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" containerID="cri-o://323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca" gracePeriod=30 Oct 30 06:39:39.234084 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:39.234054 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp"] Oct 30 06:39:39.234282 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:39.234238 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" containerID="cri-o://535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757" gracePeriod=30 Oct 30 06:39:39.566421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:39.566328 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4gvmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:39:39.577531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:39.577489 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 06:39:40.559291 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:40.559254 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4gvmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:39:40.559498 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:40.559343 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 06:39:41.992434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:41.992392 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6"] Oct 30 06:39:41.996469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:41.996441 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:42.086212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:42.086181 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:42.086818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:42.086787 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:42.088032 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:42.088010 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:39:44.526441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.526414 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:39:44.569554 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.569529 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:39:44.667590 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.667543 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-envoy\") pod \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " Oct 30 06:39:44.667777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.667665 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-data\") pod \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " Oct 30 06:39:44.667777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.667744 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-podinfo\") pod \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " Oct 30 06:39:44.667909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.667782 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-credential-socket\") pod \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " Oct 30 06:39:44.667909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.667833 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-podinfo\") pod \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " Oct 30 06:39:44.668018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.667913 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-data" pod "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" (UID: "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.668018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.667928 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-certs\") pod \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " Oct 30 06:39:44.668018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.667976 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-certs\") pod \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " Oct 30 06:39:44.668018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668013 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istiod-ca-cert\") pod \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " Oct 30 06:39:44.668225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668054 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-kube-api-access-qpb6w\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-kube-api-access-qpb6w\") pod \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " Oct 30 06:39:44.668225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668087 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-token\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-token\") pod \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " Oct 30 06:39:44.668225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668153 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-data\") pod \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " Oct 30 06:39:44.668225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668162 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-credential-socket" pod "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" (UID: "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.668225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668201 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-socket\") pod \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " Oct 30 06:39:44.668522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668290 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-socket\") pod \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " Oct 30 06:39:44.668522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668372 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-token\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-token\") pod \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " Oct 30 06:39:44.668522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668426 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istiod-ca-cert\") pod \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\" (UID: \"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b\") " Oct 30 06:39:44.668522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668461 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-kube-api-access-2vnjw\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-kube-api-access-2vnjw\") pod \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " Oct 30 06:39:44.668749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668523 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-envoy\") pod \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " Oct 30 06:39:44.668749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668618 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-credential-socket\") pod \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\" (UID: \"b6bdf7fe-250e-4998-90ad-cea9238b3ce9\") " Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.668940 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-data" pod "b6bdf7fe-250e-4998-90ad-cea9238b3ce9" (UID: "b6bdf7fe-250e-4998-90ad-cea9238b3ce9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.669184 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-credential-socket" pod "b6bdf7fe-250e-4998-90ad-cea9238b3ce9" (UID: "b6bdf7fe-250e-4998-90ad-cea9238b3ce9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.669202 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-certs" pod "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" (UID: "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.669396 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-certs" pod "b6bdf7fe-250e-4998-90ad-cea9238b3ce9" (UID: "b6bdf7fe-250e-4998-90ad-cea9238b3ce9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.669407 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-socket" pod "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" (UID: "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.669619 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-socket" pod "b6bdf7fe-250e-4998-90ad-cea9238b3ce9" (UID: "b6bdf7fe-250e-4998-90ad-cea9238b3ce9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670169 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istiod-ca-cert" pod "b6bdf7fe-250e-4998-90ad-cea9238b3ce9" (UID: "b6bdf7fe-250e-4998-90ad-cea9238b3ce9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670387 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-envoy" pod "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" (UID: "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670641 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-podinfo" pod "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" (UID: "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670730 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istiod-ca-cert" pod "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" (UID: "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670801 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670824 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670836 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670850 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670863 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670875 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670889 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670901 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670914 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.670926 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.671089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.671050 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-kube-api-access-qpb6w" pod "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" (UID: "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b"). InnerVolumeSpecName "kube-api-access-qpb6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:44.672276 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.672189 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-token" pod "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" (UID: "dcee06a3-8322-4bc6-86bf-d2e346f1fa9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:44.672609 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.672587 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-envoy" pod "b6bdf7fe-250e-4998-90ad-cea9238b3ce9" (UID: "b6bdf7fe-250e-4998-90ad-cea9238b3ce9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:39:44.673630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.673608 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-token" pod "b6bdf7fe-250e-4998-90ad-cea9238b3ce9" (UID: "b6bdf7fe-250e-4998-90ad-cea9238b3ce9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:44.674436 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.674409 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-podinfo" pod "b6bdf7fe-250e-4998-90ad-cea9238b3ce9" (UID: "b6bdf7fe-250e-4998-90ad-cea9238b3ce9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:39:44.675195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.675176 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-kube-api-access-2vnjw" pod "b6bdf7fe-250e-4998-90ad-cea9238b3ce9" (UID: "b6bdf7fe-250e-4998-90ad-cea9238b3ce9"). InnerVolumeSpecName "kube-api-access-2vnjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:44.772981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.772897 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpb6w\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-kube-api-access-qpb6w\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.772981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.772930 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.772981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.772942 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.772981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.772952 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.772981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.772967 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vnjw\" (UniqueName: \"kubernetes.io/projected/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-kube-api-access-2vnjw\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.772981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.772979 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.773350 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.772991 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bdf7fe-250e-4998-90ad-cea9238b3ce9-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.773350 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.773003 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:44.956141 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.956110 4431 generic.go:356] "Generic (PLEG): container finished" podID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerID="323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca" exitCode=0 Oct 30 06:39:44.956332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.956177 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" Oct 30 06:39:44.956332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.956194 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" event={"ID":"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b","Type":"ContainerDied","Data":"323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca"} Oct 30 06:39:44.956488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.956340 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2" event={"ID":"dcee06a3-8322-4bc6-86bf-d2e346f1fa9b","Type":"ContainerDied","Data":"2d3c8a5416035601acbc77cde44fd23ba6df954ae722838f47033c5a29325afb"} Oct 30 06:39:44.956488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.956360 4431 scope.go:122] "RemoveContainer" containerID="323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca" Oct 30 06:39:44.958556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.958535 4431 generic.go:356] "Generic (PLEG): container finished" podID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerID="535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757" exitCode=0 Oct 30 06:39:44.958682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.958586 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" Oct 30 06:39:44.958682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.958605 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" event={"ID":"b6bdf7fe-250e-4998-90ad-cea9238b3ce9","Type":"ContainerDied","Data":"535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757"} Oct 30 06:39:44.958682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.958661 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp" event={"ID":"b6bdf7fe-250e-4998-90ad-cea9238b3ce9","Type":"ContainerDied","Data":"9c02bc716d27f2a944da8f7479446259a30742aaee777ee406b3bb345b6a1444"} Oct 30 06:39:44.970841 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.970826 4431 scope.go:122] "RemoveContainer" containerID="323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca" Oct 30 06:39:44.971171 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:39:44.971149 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca\": container with ID starting with 323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca not found: ID does not exist" containerID="323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca" Oct 30 06:39:44.971250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.971179 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca"} err="failed to get container status \"323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca\": rpc error: code = NotFound desc = could not find container \"323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca\": container with ID starting with 323b94bea72b0902f914d55605f2b926569f21ba2f706edfa2d0aaff2e85a2ca not found: ID does not exist" Oct 30 06:39:44.971250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.971198 4431 scope.go:122] "RemoveContainer" containerID="535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757" Oct 30 06:39:44.982848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.982823 4431 scope.go:122] "RemoveContainer" containerID="535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757" Oct 30 06:39:44.983110 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:39:44.983088 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757\": container with ID starting with 535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757 not found: ID does not exist" containerID="535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757" Oct 30 06:39:44.983156 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:44.983120 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757"} err="failed to get container status \"535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757\": rpc error: code = NotFound desc = could not find container \"535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757\": container with ID starting with 535232ef7d151126b9ea53eff683def926bc0481620b7d1f96f4a7747d0d6757 not found: ID does not exist" Oct 30 06:39:45.019101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:45.019079 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2"] Oct 30 06:39:45.023813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:45.023758 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-58ms2"] Oct 30 06:39:45.063757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:45.063730 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp"] Oct 30 06:39:45.067245 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:45.067224 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-4mblp"] Oct 30 06:39:45.339942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:45.339873 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" path="/var/lib/kubelet/pods/b6bdf7fe-250e-4998-90ad-cea9238b3ce9/volumes" Oct 30 06:39:45.340467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:45.340451 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" path="/var/lib/kubelet/pods/dcee06a3-8322-4bc6-86bf-d2e346f1fa9b/volumes" Oct 30 06:39:46.992923 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:46.992874 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" podUID="3ef5f952-d068-4817-9f8d-d76642d13d40" containerName="payload-processing" containerID="cri-o://4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50" gracePeriod=25 Oct 30 06:39:47.256692 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.256632 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:47.417136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.415262 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-provider-credentials\") pod \"3ef5f952-d068-4817-9f8d-d76642d13d40\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " Oct 30 06:39:47.417136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.415475 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-kube-api-access-lxx4r\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-kube-api-access-lxx4r\") pod \"3ef5f952-d068-4817-9f8d-d76642d13d40\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " Oct 30 06:39:47.417136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.415616 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-routing-overlay-volume\") pod \"3ef5f952-d068-4817-9f8d-d76642d13d40\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " Oct 30 06:39:47.417136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.415650 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-plugins-config-volume\") pod \"3ef5f952-d068-4817-9f8d-d76642d13d40\" (UID: \"3ef5f952-d068-4817-9f8d-d76642d13d40\") " Oct 30 06:39:47.419605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.417966 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-plugins-config-volume" pod "3ef5f952-d068-4817-9f8d-d76642d13d40" (UID: "3ef5f952-d068-4817-9f8d-d76642d13d40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:47.419605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.418368 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-routing-overlay-volume" pod "3ef5f952-d068-4817-9f8d-d76642d13d40" (UID: "3ef5f952-d068-4817-9f8d-d76642d13d40"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:39:47.422106 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.421829 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-kube-api-access-lxx4r" pod "3ef5f952-d068-4817-9f8d-d76642d13d40" (UID: "3ef5f952-d068-4817-9f8d-d76642d13d40"). InnerVolumeSpecName "kube-api-access-lxx4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:47.424845 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.424776 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-provider-credentials" pod "3ef5f952-d068-4817-9f8d-d76642d13d40" (UID: "3ef5f952-d068-4817-9f8d-d76642d13d40"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:39:47.517597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.517523 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:47.517597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.517549 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxx4r\" (UniqueName: \"kubernetes.io/projected/3ef5f952-d068-4817-9f8d-d76642d13d40-kube-api-access-lxx4r\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:47.517597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.517559 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:47.517597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.517567 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef5f952-d068-4817-9f8d-d76642d13d40-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:39:47.982988 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.982957 4431 generic.go:356] "Generic (PLEG): container finished" podID="3ef5f952-d068-4817-9f8d-d76642d13d40" containerID="4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50" exitCode=0 Oct 30 06:39:47.983153 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.983004 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" Oct 30 06:39:47.983153 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.983038 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" event={"ID":"3ef5f952-d068-4817-9f8d-d76642d13d40","Type":"ContainerDied","Data":"4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50"} Oct 30 06:39:47.983153 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.983088 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6" event={"ID":"3ef5f952-d068-4817-9f8d-d76642d13d40","Type":"ContainerDied","Data":"cc5dbc704fee5d0f145de2d39ab409df45358447f8a47ce089474cf272be10ef"} Oct 30 06:39:47.983153 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.983110 4431 scope.go:122] "RemoveContainer" containerID="4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50" Oct 30 06:39:47.993400 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.993379 4431 scope.go:122] "RemoveContainer" containerID="4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50" Oct 30 06:39:47.993704 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:39:47.993639 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50\": container with ID starting with 4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50 not found: ID does not exist" containerID="4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50" Oct 30 06:39:47.993704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:47.993663 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50"} err="failed to get container status \"4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50\": rpc error: code = NotFound desc = could not find container \"4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50\": container with ID starting with 4a3286f74a0b13c5b75d674ca8e11d630309179d1dee14324bd43b2687face50 not found: ID does not exist" Oct 30 06:39:48.037272 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:48.037243 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6"] Oct 30 06:39:48.047275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:48.047251 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-5f6b5prq6"] Oct 30 06:39:49.339514 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:49.339485 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef5f952-d068-4817-9f8d-d76642d13d40" path="/var/lib/kubelet/pods/3ef5f952-d068-4817-9f8d-d76642d13d40/volumes" Oct 30 06:39:59.811441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.811363 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7"] Oct 30 06:39:59.831229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.831195 4431 cpu_manager.go:414] "containerMap: removing container" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" Oct 30 06:39:59.831491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.831466 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" Oct 30 06:39:59.831655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.831635 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3ef5f952-d068-4817-9f8d-d76642d13d40" containerName="payload-processing" Oct 30 06:39:59.831744 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.831733 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef5f952-d068-4817-9f8d-d76642d13d40" containerName="payload-processing" Oct 30 06:39:59.831863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.831853 4431 cpu_manager.go:414] "containerMap: removing container" podUID="62be59ea-4ef4-4a67-b521-16cba75dd82c" containerName="payload-processing" Oct 30 06:39:59.831931 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.831922 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62be59ea-4ef4-4a67-b521-16cba75dd82c" containerName="payload-processing" Oct 30 06:39:59.832020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.832011 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" Oct 30 06:39:59.832088 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.832078 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" Oct 30 06:39:59.832380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.832365 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef5f952-d068-4817-9f8d-d76642d13d40" containerName="payload-processing" Oct 30 06:39:59.832491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.832481 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="62be59ea-4ef4-4a67-b521-16cba75dd82c" containerName="payload-processing" Oct 30 06:39:59.832567 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.832559 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bdf7fe-250e-4998-90ad-cea9238b3ce9" containerName="istio-proxy" Oct 30 06:39:59.832656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.832645 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcee06a3-8322-4bc6-86bf-d2e346f1fa9b" containerName="istio-proxy" Oct 30 06:39:59.848320 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.838554 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:39:59.917324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.908800 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"openshift-service-ca.crt\"" Oct 30 06:39:59.917324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.909017 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"routing-overlay\"" Oct 30 06:39:59.917324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.909080 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f0145f\"" Oct 30 06:39:59.917324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.909166 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"kube-root-ca.crt\"" Oct 30 06:39:59.917324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.909233 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"agc-cred-338c7053d51e\"" Oct 30 06:39:59.917324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:39:59.909969 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f0145f\"/\"payload-processing-external-model-e2e--7b857ee5-dockercfg-fx57l\"" Oct 30 06:40:00.056342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.047522 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.056342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.047598 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.056342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.047666 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.056342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.047740 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw55l\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-kube-api-access-sw55l\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.097379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.093514 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7"] Oct 30 06:40:00.175378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.156427 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.175378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.156499 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.175378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.156569 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.175378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.156634 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw55l\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-kube-api-access-sw55l\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.175378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.158282 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.175378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.162143 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.175378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.168855 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.192003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.191892 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw55l\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-kube-api-access-sw55l\") pod \"payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.412713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.412630 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:00.721758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:00.721726 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7"] Oct 30 06:40:00.723112 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:40:00.723086 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f9fa49_0752_4453_bf1f_f30dac153ef4.slice/crio-da8e58ad547d57f606187bcdbc84b1bc4b8acaf2b665f958517fe6da4a87f4b1 WatchSource:0}: Error finding container da8e58ad547d57f606187bcdbc84b1bc4b8acaf2b665f958517fe6da4a87f4b1: Status 404 returned error can't find the container with id da8e58ad547d57f606187bcdbc84b1bc4b8acaf2b665f958517fe6da4a87f4b1 Oct 30 06:40:01.190348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:01.190253 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" event={"ID":"04f9fa49-0752-4453-bf1f-f30dac153ef4","Type":"ContainerStarted","Data":"13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f"} Oct 30 06:40:01.190348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:01.190343 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" event={"ID":"04f9fa49-0752-4453-bf1f-f30dac153ef4","Type":"ContainerStarted","Data":"da8e58ad547d57f606187bcdbc84b1bc4b8acaf2b665f958517fe6da4a87f4b1"} Oct 30 06:40:01.190771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:01.190374 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:01.231140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:01.231093 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" podStartSLOduration=2.231076061 podStartE2EDuration="2.231076061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:40:01.230643434 +0000 UTC m=+2810.037165205" watchObservedRunningTime="2026-10-30 06:40:01.231076061 +0000 UTC m=+2810.037597809" Oct 30 06:40:03.701877 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:03.701843 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:40:11.203374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.203338 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f"] Oct 30 06:40:11.208889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.208871 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.225466 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.225443 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f"] Oct 30 06:40:11.233295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.233270 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"agc-cred-d2f0007faba4\"" Oct 30 06:40:11.233511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.233487 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"openshift-service-ca.crt\"" Oct 30 06:40:11.233635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.233541 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"routing-overlay\"" Oct 30 06:40:11.233635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.233628 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88acce\"" Oct 30 06:40:11.233762 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.233713 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-e2e--4e48ea35-dockercfg-6kfcg\"" Oct 30 06:40:11.235363 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.235343 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"kube-root-ca.crt\"" Oct 30 06:40:11.316924 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.316869 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.317134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.316947 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.317134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.317058 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-kube-api-access-jqgql\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.317444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.317411 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.427324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.426747 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.427324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.426938 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.427324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.426972 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.427324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.427063 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-kube-api-access-jqgql\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.509381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.508662 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88acce\"" Oct 30 06:40:11.509381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.508899 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"routing-overlay\"" Oct 30 06:40:11.517260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.517227 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.518260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.518235 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.534633 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.534601 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"kube-root-ca.crt\"" Oct 30 06:40:11.556175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.556103 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.557236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.557205 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"openshift-service-ca.crt\"" Oct 30 06:40:11.570181 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.570127 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-kube-api-access-jqgql\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:11.838367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.838278 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-e2e--4e48ea35-dockercfg-6kfcg\"" Oct 30 06:40:11.844804 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:11.844778 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:12.046388 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:12.046358 4431 scope.go:122] "RemoveContainer" containerID="972c2b089a145348a713a5824e37cbf4ea3a46301fbbc1461ba506d17cef702c" Oct 30 06:40:12.153409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:12.153380 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f"] Oct 30 06:40:12.153645 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:40:12.153620 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7254a8_3671_48a5_a640_5f42a1e40ca7.slice/crio-b336e4c85987d1904ae21250f8fbdccc3f2c72836b5fd5b688ed6a9548a90571 WatchSource:0}: Error finding container b336e4c85987d1904ae21250f8fbdccc3f2c72836b5fd5b688ed6a9548a90571: Status 404 returned error can't find the container with id b336e4c85987d1904ae21250f8fbdccc3f2c72836b5fd5b688ed6a9548a90571 Oct 30 06:40:12.201187 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:12.201164 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:40:12.362467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:12.362389 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" event={"ID":"3e7254a8-3671-48a5-a640-5f42a1e40ca7","Type":"ContainerStarted","Data":"f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6"} Oct 30 06:40:12.362467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:12.362440 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" event={"ID":"3e7254a8-3671-48a5-a640-5f42a1e40ca7","Type":"ContainerStarted","Data":"b336e4c85987d1904ae21250f8fbdccc3f2c72836b5fd5b688ed6a9548a90571"} Oct 30 06:40:12.362877 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:12.362585 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:12.404472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:12.404421 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" podStartSLOduration=1.404405752 podStartE2EDuration="1.404405752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:40:12.402975462 +0000 UTC m=+2821.209497211" watchObservedRunningTime="2026-10-30 06:40:12.404405752 +0000 UTC m=+2821.210927499" Oct 30 06:40:13.495422 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:13.495391 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-d2f0007faba4: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:13.495422 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:13.495414 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:13.495908 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:13.495475 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials podName:3e7254a8-3671-48a5-a640-5f42a1e40ca7 nodeName:}" failed. No retries permitted until 2026-10-30 06:40:13.99545652 +0000 UTC m=+2822.801978245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7") : secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:13.682236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.682204 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8"] Oct 30 06:40:13.686761 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.686740 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.702497 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.702470 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"e2e-cfl-7c19a2-provider-cred\"" Oct 30 06:40:13.705868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.705844 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8"] Oct 30 06:40:13.800019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.799952 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.800155 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.800101 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.800212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.800195 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txhw\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-kube-api-access-9txhw\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.800252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.800239 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.901213 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.901177 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.901427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.901294 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.901427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.901396 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txhw\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-kube-api-access-9txhw\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.901564 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.901430 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.902049 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.902030 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.902099 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.902048 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.903676 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.903649 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:13.924550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:13.924527 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txhw\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-kube-api-access-9txhw\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:14.008620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:14.008579 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:14.010484 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:14.010455 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-d2f0007faba4: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:14.010769 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:14.010756 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:14.010947 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:14.010937 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials podName:3e7254a8-3671-48a5-a640-5f42a1e40ca7 nodeName:}" failed. No retries permitted until 2026-10-30 06:40:15.010913496 +0000 UTC m=+2823.817435242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7") : secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:14.366226 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:14.366198 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8"] Oct 30 06:40:14.366553 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:40:14.366525 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e36330_7d24_4e2c_8b1d_35e20c2ab579.slice/crio-2cc4b58fb1102752e88bd722cd2d023b31e59c8a717392a779879ae4726682db WatchSource:0}: Error finding container 2cc4b58fb1102752e88bd722cd2d023b31e59c8a717392a779879ae4726682db: Status 404 returned error can't find the container with id 2cc4b58fb1102752e88bd722cd2d023b31e59c8a717392a779879ae4726682db Oct 30 06:40:14.375593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:14.375557 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" event={"ID":"74e36330-7d24-4e2c-8b1d-35e20c2ab579","Type":"ContainerStarted","Data":"2cc4b58fb1102752e88bd722cd2d023b31e59c8a717392a779879ae4726682db"} Oct 30 06:40:15.025752 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:15.025721 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-d2f0007faba4: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:15.025752 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:15.025750 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:15.026160 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:15.025811 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials podName:3e7254a8-3671-48a5-a640-5f42a1e40ca7 nodeName:}" failed. No retries permitted until 2026-10-30 06:40:17.025794816 +0000 UTC m=+2825.832316543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7") : secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:15.384455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:15.384370 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" event={"ID":"74e36330-7d24-4e2c-8b1d-35e20c2ab579","Type":"ContainerStarted","Data":"82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999"} Oct 30 06:40:15.384455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:15.384432 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:15.426156 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:15.426112 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" podStartSLOduration=2.42609763 podStartE2EDuration="2.42609763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:40:15.425594013 +0000 UTC m=+2824.232115775" watchObservedRunningTime="2026-10-30 06:40:15.42609763 +0000 UTC m=+2824.232619379" Oct 30 06:40:17.062915 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:17.062884 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-d2f0007faba4: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:17.062915 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:17.062908 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:17.063375 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:17.062977 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials podName:3e7254a8-3671-48a5-a640-5f42a1e40ca7 nodeName:}" failed. No retries permitted until 2026-10-30 06:40:21.062956405 +0000 UTC m=+2829.869478149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7") : secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:21.117693 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:21.117658 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-d2f0007faba4: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:21.117693 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:21.117681 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:21.118128 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:21.117732 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials podName:3e7254a8-3671-48a5-a640-5f42a1e40ca7 nodeName:}" failed. No retries permitted until 2026-10-30 06:40:29.117718002 +0000 UTC m=+2837.924239742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7") : secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:23.371959 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:23.371925 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:26.416167 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:26.416089 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:40:26.494084 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:26.494046 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f"] Oct 30 06:40:29.128463 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:29.128432 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-d2f0007faba4: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:29.128463 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:29.128457 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f: secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:29.128965 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:29.128512 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials podName:3e7254a8-3671-48a5-a640-5f42a1e40ca7 nodeName:}" failed. No retries permitted until 2026-10-30 06:40:45.128494714 +0000 UTC m=+2853.935016442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7") : secret "agc-cred-d2f0007faba4" not found Oct 30 06:40:31.495362 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.495277 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" podUID="3e7254a8-3671-48a5-a640-5f42a1e40ca7" containerName="payload-processing" containerID="cri-o://f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6" gracePeriod=25 Oct 30 06:40:31.820250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.820228 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:31.866559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.866530 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-routing-overlay-volume\") pod \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " Oct 30 06:40:31.866715 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.866699 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials\") pod \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " Oct 30 06:40:31.866819 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.866806 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-kube-api-access-jqgql\") pod \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " Oct 30 06:40:31.866970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.866940 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-routing-overlay-volume" pod "3e7254a8-3671-48a5-a640-5f42a1e40ca7" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:40:31.867101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.866992 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-plugins-config-volume\") pod \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\" (UID: \"3e7254a8-3671-48a5-a640-5f42a1e40ca7\") " Oct 30 06:40:31.867293 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.867274 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-plugins-config-volume" pod "3e7254a8-3671-48a5-a640-5f42a1e40ca7" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:40:31.867555 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.867538 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:40:31.867610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.867559 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7254a8-3671-48a5-a640-5f42a1e40ca7-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:40:31.869636 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.869609 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-kube-api-access-jqgql" pod "3e7254a8-3671-48a5-a640-5f42a1e40ca7" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7"). InnerVolumeSpecName "kube-api-access-jqgql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:40:31.869858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.869840 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials" pod "3e7254a8-3671-48a5-a640-5f42a1e40ca7" (UID: "3e7254a8-3671-48a5-a640-5f42a1e40ca7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:40:31.973495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.973460 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:40:31.974142 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:31.974120 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgql\" (UniqueName: \"kubernetes.io/projected/3e7254a8-3671-48a5-a640-5f42a1e40ca7-kube-api-access-jqgql\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:40:32.533401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:32.533370 4431 generic.go:356] "Generic (PLEG): container finished" podID="3e7254a8-3671-48a5-a640-5f42a1e40ca7" containerID="f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6" exitCode=0 Oct 30 06:40:32.533954 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:32.533415 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" Oct 30 06:40:32.533954 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:32.533435 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" event={"ID":"3e7254a8-3671-48a5-a640-5f42a1e40ca7","Type":"ContainerDied","Data":"f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6"} Oct 30 06:40:32.533954 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:32.533484 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f" event={"ID":"3e7254a8-3671-48a5-a640-5f42a1e40ca7","Type":"ContainerDied","Data":"b336e4c85987d1904ae21250f8fbdccc3f2c72836b5fd5b688ed6a9548a90571"} Oct 30 06:40:32.533954 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:32.533508 4431 scope.go:122] "RemoveContainer" containerID="f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6" Oct 30 06:40:32.547031 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:32.547014 4431 scope.go:122] "RemoveContainer" containerID="f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6" Oct 30 06:40:32.547332 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:32.547279 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6\": container with ID starting with f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6 not found: ID does not exist" containerID="f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6" Oct 30 06:40:32.547411 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:32.547323 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6"} err="failed to get container status \"f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6\": rpc error: code = NotFound desc = could not find container \"f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6\": container with ID starting with f45beabbe08949f129b17ec98d5c32dd1d4ed784f66af2fe547b6a2ca1cec5b6 not found: ID does not exist" Oct 30 06:40:32.586445 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:32.586424 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f"] Oct 30 06:40:32.598063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:32.598038 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-6b878hr8f"] Oct 30 06:40:33.420350 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:40:33.416687 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7254a8-3671-48a5-a640-5f42a1e40ca7" path="/var/lib/kubelet/pods/3e7254a8-3671-48a5-a640-5f42a1e40ca7/volumes" Oct 30 06:40:33.707895 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:40:33.707854 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:41:00.998885 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:00.998847 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq"] Oct 30 06:41:03.702491 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:41:03.702455 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:41:05.999684 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:05.999640 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" podUID="7ad444b0-05db-4a98-83c7-54372ae0caa7" containerName="payload-processing" containerID="cri-o://b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8" gracePeriod=25 Oct 30 06:41:06.321648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.321620 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:41:06.393162 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.393126 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-provider-credentials\") pod \"7ad444b0-05db-4a98-83c7-54372ae0caa7\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " Oct 30 06:41:06.393440 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.393412 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-kube-api-access-4r4r5\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-kube-api-access-4r4r5\") pod \"7ad444b0-05db-4a98-83c7-54372ae0caa7\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " Oct 30 06:41:06.393598 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.393488 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-routing-overlay-volume\") pod \"7ad444b0-05db-4a98-83c7-54372ae0caa7\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " Oct 30 06:41:06.393598 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.393558 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-plugins-config-volume\") pod \"7ad444b0-05db-4a98-83c7-54372ae0caa7\" (UID: \"7ad444b0-05db-4a98-83c7-54372ae0caa7\") " Oct 30 06:41:06.393866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.393842 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-routing-overlay-volume" pod "7ad444b0-05db-4a98-83c7-54372ae0caa7" (UID: "7ad444b0-05db-4a98-83c7-54372ae0caa7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:06.394024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.393997 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-plugins-config-volume" pod "7ad444b0-05db-4a98-83c7-54372ae0caa7" (UID: "7ad444b0-05db-4a98-83c7-54372ae0caa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:06.394184 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.394168 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:06.394254 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.394185 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad444b0-05db-4a98-83c7-54372ae0caa7-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:06.395811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.395788 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-kube-api-access-4r4r5" pod "7ad444b0-05db-4a98-83c7-54372ae0caa7" (UID: "7ad444b0-05db-4a98-83c7-54372ae0caa7"). InnerVolumeSpecName "kube-api-access-4r4r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:06.395901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.395785 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-provider-credentials" pod "7ad444b0-05db-4a98-83c7-54372ae0caa7" (UID: "7ad444b0-05db-4a98-83c7-54372ae0caa7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:06.496885 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.496848 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r4r5\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-kube-api-access-4r4r5\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:06.496885 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.496887 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ad444b0-05db-4a98-83c7-54372ae0caa7-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:06.774826 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.774795 4431 generic.go:356] "Generic (PLEG): container finished" podID="7ad444b0-05db-4a98-83c7-54372ae0caa7" containerID="b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8" exitCode=0 Oct 30 06:41:06.775003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.774855 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" Oct 30 06:41:06.775003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.774873 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" event={"ID":"7ad444b0-05db-4a98-83c7-54372ae0caa7","Type":"ContainerDied","Data":"b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8"} Oct 30 06:41:06.775003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.774951 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq" event={"ID":"7ad444b0-05db-4a98-83c7-54372ae0caa7","Type":"ContainerDied","Data":"59c7959f85de5c7f796de8718eea8345e763cadbe7b6afd6df14655475ea3e96"} Oct 30 06:41:06.775003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.774975 4431 scope.go:122] "RemoveContainer" containerID="b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8" Oct 30 06:41:06.785766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.785738 4431 scope.go:122] "RemoveContainer" containerID="b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8" Oct 30 06:41:06.786061 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:41:06.786044 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8\": container with ID starting with b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8 not found: ID does not exist" containerID="b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8" Oct 30 06:41:06.786114 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.786070 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8"} err="failed to get container status \"b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8\": rpc error: code = NotFound desc = could not find container \"b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8\": container with ID starting with b3da6b9b1027762d2d72e397bd198bd42ed1d8728d88eee584f081a8899080e8 not found: ID does not exist" Oct 30 06:41:06.827678 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.827637 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq"] Oct 30 06:41:06.839414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:06.839386 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b8ca49/payload-processing-external-model-e2e-extmod-a-b8ca49-74845gngq"] Oct 30 06:41:07.339199 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:07.339165 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad444b0-05db-4a98-83c7-54372ae0caa7" path="/var/lib/kubelet/pods/7ad444b0-05db-4a98-83c7-54372ae0caa7/volumes" Oct 30 06:41:27.568951 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:27.568914 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7"] Oct 30 06:41:28.561323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:28.560948 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8"] Oct 30 06:41:32.569944 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:32.569872 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" podUID="04f9fa49-0752-4453-bf1f-f30dac153ef4" containerName="payload-processing" containerID="cri-o://13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f" gracePeriod=25 Oct 30 06:41:32.872905 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:32.872879 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:41:33.043531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.043498 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-provider-credentials\") pod \"04f9fa49-0752-4453-bf1f-f30dac153ef4\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " Oct 30 06:41:33.043728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.043614 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-plugins-config-volume\") pod \"04f9fa49-0752-4453-bf1f-f30dac153ef4\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " Oct 30 06:41:33.043728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.043713 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-routing-overlay-volume\") pod \"04f9fa49-0752-4453-bf1f-f30dac153ef4\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " Oct 30 06:41:33.043946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.043928 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-kube-api-access-sw55l\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-kube-api-access-sw55l\") pod \"04f9fa49-0752-4453-bf1f-f30dac153ef4\" (UID: \"04f9fa49-0752-4453-bf1f-f30dac153ef4\") " Oct 30 06:41:33.044031 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.044006 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-plugins-config-volume" pod "04f9fa49-0752-4453-bf1f-f30dac153ef4" (UID: "04f9fa49-0752-4453-bf1f-f30dac153ef4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:33.044095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.044079 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-routing-overlay-volume" pod "04f9fa49-0752-4453-bf1f-f30dac153ef4" (UID: "04f9fa49-0752-4453-bf1f-f30dac153ef4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:33.044432 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.044412 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:33.044510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.044436 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04f9fa49-0752-4453-bf1f-f30dac153ef4-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:33.045818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.045796 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-provider-credentials" pod "04f9fa49-0752-4453-bf1f-f30dac153ef4" (UID: "04f9fa49-0752-4453-bf1f-f30dac153ef4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:33.045904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.045888 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-kube-api-access-sw55l" pod "04f9fa49-0752-4453-bf1f-f30dac153ef4" (UID: "04f9fa49-0752-4453-bf1f-f30dac153ef4"). InnerVolumeSpecName "kube-api-access-sw55l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:33.105278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.105213 4431 generic.go:356] "Generic (PLEG): container finished" podID="04f9fa49-0752-4453-bf1f-f30dac153ef4" containerID="13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f" exitCode=0 Oct 30 06:41:33.105278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.105265 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" Oct 30 06:41:33.105278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.105272 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" event={"ID":"04f9fa49-0752-4453-bf1f-f30dac153ef4","Type":"ContainerDied","Data":"13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f"} Oct 30 06:41:33.105505 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.105376 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7" event={"ID":"04f9fa49-0752-4453-bf1f-f30dac153ef4","Type":"ContainerDied","Data":"da8e58ad547d57f606187bcdbc84b1bc4b8acaf2b665f958517fe6da4a87f4b1"} Oct 30 06:41:33.105505 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.105401 4431 scope.go:122] "RemoveContainer" containerID="13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f" Oct 30 06:41:33.115642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.115628 4431 scope.go:122] "RemoveContainer" containerID="13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f" Oct 30 06:41:33.115886 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:41:33.115868 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f\": container with ID starting with 13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f not found: ID does not exist" containerID="13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f" Oct 30 06:41:33.115929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.115896 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f"} err="failed to get container status \"13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f\": rpc error: code = NotFound desc = could not find container \"13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f\": container with ID starting with 13c3e5fa080daf2f266320064cdc09fc7065ab57b8b6699f515e9d7d3650c84f not found: ID does not exist" Oct 30 06:41:33.145441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.145419 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw55l\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-kube-api-access-sw55l\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:33.145441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.145441 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04f9fa49-0752-4453-bf1f-f30dac153ef4-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:33.156852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.156828 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7"] Oct 30 06:41:33.162696 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.162676 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f0145f/payload-processing-external-model-e2e-extmod-a-f0145f-588bf2hk7"] Oct 30 06:41:33.339259 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.339228 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f9fa49-0752-4453-bf1f-f30dac153ef4" path="/var/lib/kubelet/pods/04f9fa49-0752-4453-bf1f-f30dac153ef4/volumes" Oct 30 06:41:33.562041 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.561999 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" podUID="74e36330-7d24-4e2c-8b1d-35e20c2ab579" containerName="payload-processing" containerID="cri-o://82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999" gracePeriod=25 Oct 30 06:41:33.707263 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:41:33.707222 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:41:33.852018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.851991 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:41:33.853514 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.853489 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-routing-overlay-volume\") pod \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " Oct 30 06:41:33.853652 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.853523 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-plugins-config-volume\") pod \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " Oct 30 06:41:33.853724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.853656 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-provider-credentials\") pod \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " Oct 30 06:41:33.853792 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.853773 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-kube-api-access-9txhw\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-kube-api-access-9txhw\") pod \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\" (UID: \"74e36330-7d24-4e2c-8b1d-35e20c2ab579\") " Oct 30 06:41:33.854030 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.854005 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-plugins-config-volume" pod "74e36330-7d24-4e2c-8b1d-35e20c2ab579" (UID: "74e36330-7d24-4e2c-8b1d-35e20c2ab579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:33.854161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.854144 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-routing-overlay-volume" pod "74e36330-7d24-4e2c-8b1d-35e20c2ab579" (UID: "74e36330-7d24-4e2c-8b1d-35e20c2ab579"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:33.854461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.854445 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:33.854536 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.854463 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e36330-7d24-4e2c-8b1d-35e20c2ab579-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:33.856025 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.855992 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-kube-api-access-9txhw" pod "74e36330-7d24-4e2c-8b1d-35e20c2ab579" (UID: "74e36330-7d24-4e2c-8b1d-35e20c2ab579"). InnerVolumeSpecName "kube-api-access-9txhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:33.856350 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.856329 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-provider-credentials" pod "74e36330-7d24-4e2c-8b1d-35e20c2ab579" (UID: "74e36330-7d24-4e2c-8b1d-35e20c2ab579"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:33.955404 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.955375 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9txhw\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-kube-api-access-9txhw\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:33.955404 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:33.955403 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e36330-7d24-4e2c-8b1d-35e20c2ab579-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:34.112811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:34.112731 4431 generic.go:356] "Generic (PLEG): container finished" podID="74e36330-7d24-4e2c-8b1d-35e20c2ab579" containerID="82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999" exitCode=0 Oct 30 06:41:34.112811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:34.112771 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" Oct 30 06:41:34.112981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:34.112811 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" event={"ID":"74e36330-7d24-4e2c-8b1d-35e20c2ab579","Type":"ContainerDied","Data":"82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999"} Oct 30 06:41:34.112981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:34.112863 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8" event={"ID":"74e36330-7d24-4e2c-8b1d-35e20c2ab579","Type":"ContainerDied","Data":"2cc4b58fb1102752e88bd722cd2d023b31e59c8a717392a779879ae4726682db"} Oct 30 06:41:34.112981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:34.112886 4431 scope.go:122] "RemoveContainer" containerID="82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999" Oct 30 06:41:34.124089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:34.124071 4431 scope.go:122] "RemoveContainer" containerID="82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999" Oct 30 06:41:34.124356 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:41:34.124325 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999\": container with ID starting with 82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999 not found: ID does not exist" containerID="82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999" Oct 30 06:41:34.124451 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:34.124364 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999"} err="failed to get container status \"82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999\": rpc error: code = NotFound desc = could not find container \"82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999\": container with ID starting with 82a6b283023a4c0378f85a28ed09fa6a51afa4d4bab8a870364041f7f8a13999 not found: ID does not exist" Oct 30 06:41:34.164380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:34.164341 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8"] Oct 30 06:41:34.176078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:34.176058 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-c67fdbmm8"] Oct 30 06:41:35.340837 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:35.340804 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e36330-7d24-4e2c-8b1d-35e20c2ab579" path="/var/lib/kubelet/pods/74e36330-7d24-4e2c-8b1d-35e20c2ab579/volumes" Oct 30 06:41:40.512413 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:40.512382 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7"] Oct 30 06:41:41.247798 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:41.247766 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65"] Oct 30 06:41:41.248020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:41.247997 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" containerID="cri-o://3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5" gracePeriod=30 Oct 30 06:41:45.513165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.513121 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" podUID="9694460e-38ab-48bb-9f72-824a44c171fe" containerName="payload-processing" containerID="cri-o://a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44" gracePeriod=25 Oct 30 06:41:45.853321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.847167 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:41:45.929874 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.929840 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-routing-overlay-volume\") pod \"9694460e-38ab-48bb-9f72-824a44c171fe\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " Oct 30 06:41:45.930061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.929968 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-kube-api-access-6q9bl\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-kube-api-access-6q9bl\") pod \"9694460e-38ab-48bb-9f72-824a44c171fe\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " Oct 30 06:41:45.930061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.930037 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-provider-credentials\") pod \"9694460e-38ab-48bb-9f72-824a44c171fe\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " Oct 30 06:41:45.930208 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.930189 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-plugins-config-volume\") pod \"9694460e-38ab-48bb-9f72-824a44c171fe\" (UID: \"9694460e-38ab-48bb-9f72-824a44c171fe\") " Oct 30 06:41:45.930295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.930270 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-routing-overlay-volume" pod "9694460e-38ab-48bb-9f72-824a44c171fe" (UID: "9694460e-38ab-48bb-9f72-824a44c171fe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:45.930724 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.930702 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-plugins-config-volume" pod "9694460e-38ab-48bb-9f72-824a44c171fe" (UID: "9694460e-38ab-48bb-9f72-824a44c171fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:45.930858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.930836 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:45.930930 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.930863 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9694460e-38ab-48bb-9f72-824a44c171fe-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:45.932606 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.932578 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-provider-credentials" pod "9694460e-38ab-48bb-9f72-824a44c171fe" (UID: "9694460e-38ab-48bb-9f72-824a44c171fe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:45.933012 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:45.932994 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-kube-api-access-6q9bl" pod "9694460e-38ab-48bb-9f72-824a44c171fe" (UID: "9694460e-38ab-48bb-9f72-824a44c171fe"). InnerVolumeSpecName "kube-api-access-6q9bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:46.032124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.032055 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q9bl\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-kube-api-access-6q9bl\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.032124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.032090 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9694460e-38ab-48bb-9f72-824a44c171fe-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.266784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.266756 4431 generic.go:356] "Generic (PLEG): container finished" podID="9694460e-38ab-48bb-9f72-824a44c171fe" containerID="a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44" exitCode=0 Oct 30 06:41:46.266902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.266801 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" Oct 30 06:41:46.266902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.266835 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" event={"ID":"9694460e-38ab-48bb-9f72-824a44c171fe","Type":"ContainerDied","Data":"a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44"} Oct 30 06:41:46.266902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.266888 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7" event={"ID":"9694460e-38ab-48bb-9f72-824a44c171fe","Type":"ContainerDied","Data":"e1d1b8d87747037874b27359d6fbeb85097b6bab0e81db60034e2971962b3f75"} Oct 30 06:41:46.267047 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.266913 4431 scope.go:122] "RemoveContainer" containerID="a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44" Oct 30 06:41:46.347412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.347381 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7"] Oct 30 06:41:46.359801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.359773 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d2923/payload-processing-external-model-e2e-extmod-a-2d2923-79bfn4bq7"] Oct 30 06:41:46.361664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.361645 4431 scope.go:122] "RemoveContainer" containerID="a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44" Oct 30 06:41:46.362002 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:41:46.361967 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44\": container with ID starting with a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44 not found: ID does not exist" containerID="a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44" Oct 30 06:41:46.362057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.361995 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44"} err="failed to get container status \"a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44\": rpc error: code = NotFound desc = could not find container \"a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44\": container with ID starting with a5a627699fa68fb16de49ec63d2536409b1b08dfd26a0f98a6c9d6e3d2ea8e44 not found: ID does not exist" Oct 30 06:41:46.510151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.510127 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:41:46.641552 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641471 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istiod-ca-cert\") pod \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " Oct 30 06:41:46.641552 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641511 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-envoy\") pod \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641575 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-podinfo\") pod \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641614 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-data\") pod \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641676 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-credential-socket\") pod \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641699 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-kube-api-access-b66v2\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-kube-api-access-b66v2\") pod \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641763 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-certs\") pod \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641848 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-socket\") pod \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641926 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-token\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-token\") pod \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\" (UID: \"2dca7a50-af82-4fc2-8d40-846f77af3fc0\") " Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.641915 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istiod-ca-cert" pod "2dca7a50-af82-4fc2-8d40-846f77af3fc0" (UID: "2dca7a50-af82-4fc2-8d40-846f77af3fc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.642019 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-credential-socket" pod "2dca7a50-af82-4fc2-8d40-846f77af3fc0" (UID: "2dca7a50-af82-4fc2-8d40-846f77af3fc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:46.642086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.642060 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-certs" pod "2dca7a50-af82-4fc2-8d40-846f77af3fc0" (UID: "2dca7a50-af82-4fc2-8d40-846f77af3fc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:46.642668 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.642215 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-socket" pod "2dca7a50-af82-4fc2-8d40-846f77af3fc0" (UID: "2dca7a50-af82-4fc2-8d40-846f77af3fc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:46.642668 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.642351 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-data" pod "2dca7a50-af82-4fc2-8d40-846f77af3fc0" (UID: "2dca7a50-af82-4fc2-8d40-846f77af3fc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:46.642782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.642668 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.642782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.642688 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.642782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.642697 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.642782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.642707 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.642782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.642724 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.644449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.644421 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-podinfo" pod "2dca7a50-af82-4fc2-8d40-846f77af3fc0" (UID: "2dca7a50-af82-4fc2-8d40-846f77af3fc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:41:46.644575 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.644480 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-envoy" pod "2dca7a50-af82-4fc2-8d40-846f77af3fc0" (UID: "2dca7a50-af82-4fc2-8d40-846f77af3fc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:41:46.644575 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.644542 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-kube-api-access-b66v2" pod "2dca7a50-af82-4fc2-8d40-846f77af3fc0" (UID: "2dca7a50-af82-4fc2-8d40-846f77af3fc0"). InnerVolumeSpecName "kube-api-access-b66v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:46.644695 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.644630 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-token" pod "2dca7a50-af82-4fc2-8d40-846f77af3fc0" (UID: "2dca7a50-af82-4fc2-8d40-846f77af3fc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:46.744415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.744384 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.744415 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.744411 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b66v2\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-kube-api-access-b66v2\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.744623 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.744422 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:46.744623 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:46.744431 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca7a50-af82-4fc2-8d40-846f77af3fc0-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:47.276850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.276803 4431 generic.go:356] "Generic (PLEG): container finished" podID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerID="3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5" exitCode=0 Oct 30 06:41:47.276850 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.276848 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" event={"ID":"2dca7a50-af82-4fc2-8d40-846f77af3fc0","Type":"ContainerDied","Data":"3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5"} Oct 30 06:41:47.277105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.276863 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" Oct 30 06:41:47.277105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.276907 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65" event={"ID":"2dca7a50-af82-4fc2-8d40-846f77af3fc0","Type":"ContainerDied","Data":"6bc6166d1571d56bf5dd946aff180edeb2fbaefe64b578e3ac4d95b48ca85de4"} Oct 30 06:41:47.277105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.276937 4431 scope.go:122] "RemoveContainer" containerID="3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5" Oct 30 06:41:47.289166 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.289140 4431 scope.go:122] "RemoveContainer" containerID="3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5" Oct 30 06:41:47.289463 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:41:47.289442 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5\": container with ID starting with 3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5 not found: ID does not exist" containerID="3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5" Oct 30 06:41:47.289546 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.289474 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5"} err="failed to get container status \"3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5\": rpc error: code = NotFound desc = could not find container \"3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5\": container with ID starting with 3cc00311bb9b0ab4dfc6fefa93d6b63a51b3db82943451617881b1a5c9d757b5 not found: ID does not exist" Oct 30 06:41:47.340079 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.340056 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9694460e-38ab-48bb-9f72-824a44c171fe" path="/var/lib/kubelet/pods/9694460e-38ab-48bb-9f72-824a44c171fe/volumes" Oct 30 06:41:47.343235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.343211 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65"] Oct 30 06:41:47.355247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:47.355227 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-vbs65"] Oct 30 06:41:48.361324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.360196 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt"] Oct 30 06:41:48.361324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361230 4431 cpu_manager.go:414] "containerMap: removing container" podUID="9694460e-38ab-48bb-9f72-824a44c171fe" containerName="payload-processing" Oct 30 06:41:48.361324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361251 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9694460e-38ab-48bb-9f72-824a44c171fe" containerName="payload-processing" Oct 30 06:41:48.361324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361286 4431 cpu_manager.go:414] "containerMap: removing container" podUID="04f9fa49-0752-4453-bf1f-f30dac153ef4" containerName="payload-processing" Oct 30 06:41:48.361324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361294 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f9fa49-0752-4453-bf1f-f30dac153ef4" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361344 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361353 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361385 4431 cpu_manager.go:414] "containerMap: removing container" podUID="74e36330-7d24-4e2c-8b1d-35e20c2ab579" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361392 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e36330-7d24-4e2c-8b1d-35e20c2ab579" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361404 4431 cpu_manager.go:414] "containerMap: removing container" podUID="7ad444b0-05db-4a98-83c7-54372ae0caa7" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361413 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad444b0-05db-4a98-83c7-54372ae0caa7" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361445 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3e7254a8-3671-48a5-a640-5f42a1e40ca7" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361451 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7254a8-3671-48a5-a640-5f42a1e40ca7" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361603 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e36330-7d24-4e2c-8b1d-35e20c2ab579" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361616 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad444b0-05db-4a98-83c7-54372ae0caa7" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361631 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7254a8-3671-48a5-a640-5f42a1e40ca7" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361641 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9694460e-38ab-48bb-9f72-824a44c171fe" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361654 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f9fa49-0752-4453-bf1f-f30dac153ef4" containerName="payload-processing" Oct 30 06:41:48.362083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.361665 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" containerName="istio-proxy" Oct 30 06:41:48.391333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.379281 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.392693 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.392662 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt"] Oct 30 06:41:48.427325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.419122 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 06:41:48.427325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.419341 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 06:41:48.427325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.419680 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-8z4m5\"" Oct 30 06:41:48.427325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.419759 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 06:41:48.427325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.419867 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 06:41:48.427325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.420018 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8b3017c11d13\"" Oct 30 06:41:48.427325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.420116 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cdae46b39755\"" Oct 30 06:41:48.427325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.420237 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a6e5b91b8e65\"" Oct 30 06:41:48.427325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.420349 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b5bb1149aea2\"" Oct 30 06:41:48.451425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.443765 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-routing-overlay-volume\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.451425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.443874 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-plugins-config-volume\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.451425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.443982 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-provider-credentials\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.451425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.444185 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvj2l\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-kube-api-access-gvj2l\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.555374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.545686 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-plugins-config-volume\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.555374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.545761 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-provider-credentials\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.555374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.545964 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvj2l\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-kube-api-access-gvj2l\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.555374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.546086 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-routing-overlay-volume\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.555374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.547096 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-routing-overlay-volume\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.555374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.547874 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-plugins-config-volume\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.594324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.594226 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-provider-credentials\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.630391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.621459 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvj2l\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-kube-api-access-gvj2l\") pod \"payload-processing-external-model-6969648bc7-2r5xt\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:48.786366 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:48.786286 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:49.133260 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:49.132998 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt"] Oct 30 06:41:49.134882 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:41:49.134847 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b1784f_2075_4983_a27a_15c08fcc51c1.slice/crio-c5a44323248dad74414ecb02bab2efb0e9111a603cc4207484030035849c98be WatchSource:0}: Error finding container c5a44323248dad74414ecb02bab2efb0e9111a603cc4207484030035849c98be: Status 404 returned error can't find the container with id c5a44323248dad74414ecb02bab2efb0e9111a603cc4207484030035849c98be Oct 30 06:41:49.299755 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:49.299648 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" event={"ID":"89b1784f-2075-4983-a27a-15c08fcc51c1","Type":"ContainerStarted","Data":"8067d8ccd399974d2c5fdfc00b3a3bd25bb574744ab6d86efc270f5a6d1b23b0"} Oct 30 06:41:49.300054 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:49.300022 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" event={"ID":"89b1784f-2075-4983-a27a-15c08fcc51c1","Type":"ContainerStarted","Data":"c5a44323248dad74414ecb02bab2efb0e9111a603cc4207484030035849c98be"} Oct 30 06:41:49.300175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:49.300075 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:49.360056 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:49.360014 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dca7a50-af82-4fc2-8d40-846f77af3fc0" path="/var/lib/kubelet/pods/2dca7a50-af82-4fc2-8d40-846f77af3fc0/volumes" Oct 30 06:41:53.158636 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.158582 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" podStartSLOduration=5.15856656 podStartE2EDuration="5.15856656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:49.345612691 +0000 UTC m=+2918.152134443" watchObservedRunningTime="2026-10-30 06:41:53.15856656 +0000 UTC m=+2921.965088329" Oct 30 06:41:53.161600 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.161575 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt"] Oct 30 06:41:53.195991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.195963 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p"] Oct 30 06:41:53.201705 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.201687 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.220517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.220497 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p"] Oct 30 06:41:53.301836 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.301797 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-routing-overlay-volume\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.302033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.301918 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-plugins-config-volume\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.302033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.302022 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-provider-credentials\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.302160 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.302081 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-kube-api-access-dwjds\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.415376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.409679 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-provider-credentials\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.415376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.409775 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-kube-api-access-dwjds\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.415376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.409992 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-routing-overlay-volume\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.415376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.410108 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-plugins-config-volume\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.415376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.412603 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-routing-overlay-volume\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.415376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.413538 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-plugins-config-volume\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.440566 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.440530 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-provider-credentials\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.495247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.495202 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-kube-api-access-dwjds\") pod \"payload-processing-external-model-5b5448f9b9-hx69p\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.518602 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.518563 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:53.920397 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:53.920368 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p"] Oct 30 06:41:53.923388 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:41:53.922765 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5555c9_512c_4d75_9aeb_8ec2524b9496.slice/crio-96b473bf01c1efb31c8a0de768ec941e552f87bbe3e77d32cad5a79d02f2d0cb WatchSource:0}: Error finding container 96b473bf01c1efb31c8a0de768ec941e552f87bbe3e77d32cad5a79d02f2d0cb: Status 404 returned error can't find the container with id 96b473bf01c1efb31c8a0de768ec941e552f87bbe3e77d32cad5a79d02f2d0cb Oct 30 06:41:54.360765 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:54.360726 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" event={"ID":"3e5555c9-512c-4d75-9aeb-8ec2524b9496","Type":"ContainerStarted","Data":"36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe"} Oct 30 06:41:54.361230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:54.360774 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" event={"ID":"3e5555c9-512c-4d75-9aeb-8ec2524b9496","Type":"ContainerStarted","Data":"96b473bf01c1efb31c8a0de768ec941e552f87bbe3e77d32cad5a79d02f2d0cb"} Oct 30 06:41:54.361230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:54.360949 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:41:54.411199 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:54.411147 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" podStartSLOduration=1.4111329000000001 podStartE2EDuration="1.4111329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:54.408163082 +0000 UTC m=+2923.214684843" watchObservedRunningTime="2026-10-30 06:41:54.4111329 +0000 UTC m=+2923.217654668" Oct 30 06:41:56.058325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:56.058246 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p"] Oct 30 06:41:58.165323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.164710 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" podUID="89b1784f-2075-4983-a27a-15c08fcc51c1" containerName="payload-processing" containerID="cri-o://8067d8ccd399974d2c5fdfc00b3a3bd25bb574744ab6d86efc270f5a6d1b23b0" gracePeriod=25 Oct 30 06:41:58.419910 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.419836 4431 generic.go:356] "Generic (PLEG): container finished" podID="89b1784f-2075-4983-a27a-15c08fcc51c1" containerID="8067d8ccd399974d2c5fdfc00b3a3bd25bb574744ab6d86efc270f5a6d1b23b0" exitCode=0 Oct 30 06:41:58.420112 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.420094 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" event={"ID":"89b1784f-2075-4983-a27a-15c08fcc51c1","Type":"ContainerDied","Data":"8067d8ccd399974d2c5fdfc00b3a3bd25bb574744ab6d86efc270f5a6d1b23b0"} Oct 30 06:41:58.582547 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.582523 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:58.717181 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.717148 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-plugins-config-volume\") pod \"89b1784f-2075-4983-a27a-15c08fcc51c1\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " Oct 30 06:41:58.717360 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.717206 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-provider-credentials\") pod \"89b1784f-2075-4983-a27a-15c08fcc51c1\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " Oct 30 06:41:58.717360 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.717262 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-routing-overlay-volume\") pod \"89b1784f-2075-4983-a27a-15c08fcc51c1\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " Oct 30 06:41:58.717441 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.717377 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-kube-api-access-gvj2l\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-kube-api-access-gvj2l\") pod \"89b1784f-2075-4983-a27a-15c08fcc51c1\" (UID: \"89b1784f-2075-4983-a27a-15c08fcc51c1\") " Oct 30 06:41:58.717650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.717619 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-plugins-config-volume" pod "89b1784f-2075-4983-a27a-15c08fcc51c1" (UID: "89b1784f-2075-4983-a27a-15c08fcc51c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:58.717790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.717650 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-routing-overlay-volume" pod "89b1784f-2075-4983-a27a-15c08fcc51c1" (UID: "89b1784f-2075-4983-a27a-15c08fcc51c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:41:58.718036 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.718019 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:58.718090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.718040 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89b1784f-2075-4983-a27a-15c08fcc51c1-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:58.719580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.719551 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-provider-credentials" pod "89b1784f-2075-4983-a27a-15c08fcc51c1" (UID: "89b1784f-2075-4983-a27a-15c08fcc51c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:58.719634 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.719616 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-kube-api-access-gvj2l" pod "89b1784f-2075-4983-a27a-15c08fcc51c1" (UID: "89b1784f-2075-4983-a27a-15c08fcc51c1"). InnerVolumeSpecName "kube-api-access-gvj2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:41:58.746466 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.746436 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr"] Oct 30 06:41:58.747095 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.747081 4431 cpu_manager.go:414] "containerMap: removing container" podUID="89b1784f-2075-4983-a27a-15c08fcc51c1" containerName="payload-processing" Oct 30 06:41:58.747152 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.747096 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b1784f-2075-4983-a27a-15c08fcc51c1" containerName="payload-processing" Oct 30 06:41:58.747212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.747203 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b1784f-2075-4983-a27a-15c08fcc51c1" containerName="payload-processing" Oct 30 06:41:58.751187 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.751171 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.769347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.768442 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c537891062d6\"" Oct 30 06:41:58.770503 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.770478 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr"] Oct 30 06:41:58.825650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.825615 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.829974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.825655 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gk8\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-kube-api-access-p8gk8\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.829974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.825692 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-routing-overlay-volume\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.829974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.825925 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-plugins-config-volume\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.829974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.826118 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:58.829974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.826142 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvj2l\" (UniqueName: \"kubernetes.io/projected/89b1784f-2075-4983-a27a-15c08fcc51c1-kube-api-access-gvj2l\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:41:58.927187 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.927155 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-plugins-config-volume\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.927419 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.927393 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.927525 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.927437 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gk8\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-kube-api-access-p8gk8\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.927525 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.927475 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-routing-overlay-volume\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.928072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.928050 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-plugins-config-volume\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.928072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.928063 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-routing-overlay-volume\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.930099 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.930077 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:58.950384 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:58.950360 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gk8\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-kube-api-access-p8gk8\") pod \"payload-processing-external-model-5cd7f677db-sqssr\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:59.065170 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:59.065108 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:41:59.369121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:59.369092 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr"] Oct 30 06:41:59.369881 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:41:59.369858 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4568ed0_e18c_427f_9cba_ff7c960b6d6a.slice/crio-bdc93ad8c215c4115a6f1bbc87948e6a9e4525d25298dac29eba6ac842f641b7 WatchSource:0}: Error finding container bdc93ad8c215c4115a6f1bbc87948e6a9e4525d25298dac29eba6ac842f641b7: Status 404 returned error can't find the container with id bdc93ad8c215c4115a6f1bbc87948e6a9e4525d25298dac29eba6ac842f641b7 Oct 30 06:41:59.428520 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:59.428494 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" event={"ID":"a4568ed0-e18c-427f-9cba-ff7c960b6d6a","Type":"ContainerStarted","Data":"bdc93ad8c215c4115a6f1bbc87948e6a9e4525d25298dac29eba6ac842f641b7"} Oct 30 06:41:59.429914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:59.429890 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" event={"ID":"89b1784f-2075-4983-a27a-15c08fcc51c1","Type":"ContainerDied","Data":"c5a44323248dad74414ecb02bab2efb0e9111a603cc4207484030035849c98be"} Oct 30 06:41:59.430021 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:59.429919 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt" Oct 30 06:41:59.430021 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:59.429943 4431 scope.go:122] "RemoveContainer" containerID="8067d8ccd399974d2c5fdfc00b3a3bd25bb574744ab6d86efc270f5a6d1b23b0" Oct 30 06:41:59.481509 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:59.481484 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt"] Oct 30 06:41:59.486196 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:41:59.486176 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6969648bc7-2r5xt"] Oct 30 06:42:00.438426 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:00.438390 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" event={"ID":"a4568ed0-e18c-427f-9cba-ff7c960b6d6a","Type":"ContainerStarted","Data":"ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1"} Oct 30 06:42:00.438886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:00.438614 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:42:00.481912 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:00.481864 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" podStartSLOduration=2.481851395 podStartE2EDuration="2.481851395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:42:00.478213013 +0000 UTC m=+2929.284734761" watchObservedRunningTime="2026-10-30 06:42:00.481851395 +0000 UTC m=+2929.288373143" Oct 30 06:42:01.340185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.340151 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b1784f-2075-4983-a27a-15c08fcc51c1" path="/var/lib/kubelet/pods/89b1784f-2075-4983-a27a-15c08fcc51c1/volumes" Oct 30 06:42:01.347510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.347485 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr"] Oct 30 06:42:01.375557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.375524 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" podUID="3e5555c9-512c-4d75-9aeb-8ec2524b9496" containerName="payload-processing" containerID="cri-o://36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe" gracePeriod=25 Oct 30 06:42:01.560328 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:01.553315 4431 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b5bb1149aea2: secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:01.561483 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:01.561458 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr: secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:01.561702 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:01.561687 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials podName:a4568ed0-e18c-427f-9cba-ff7c960b6d6a nodeName:}" failed. No retries permitted until 2026-10-30 06:42:02.061664179 +0000 UTC m=+2930.868185908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials") pod "payload-processing-external-model-5cd7f677db-sqssr" (UID: "a4568ed0-e18c-427f-9cba-ff7c960b6d6a") : secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:01.714228 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.712873 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk"] Oct 30 06:42:01.720108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.720075 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.765471 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.765395 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk"] Oct 30 06:42:01.880326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.875418 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.880326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.875498 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.880326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.875597 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.880326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.875673 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.880326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.875698 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.880326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.875851 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.880326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.875928 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhw66\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-kube-api-access-lhw66\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.880326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.876032 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.880326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.876069 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978004 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978148 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978214 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhw66\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-kube-api-access-lhw66\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978315 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978350 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978400 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978443 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978520 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978585 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.978988 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.979893 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.983488 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.983740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.983710 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.987151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.987057 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.987526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.987502 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:01.987805 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:01.987782 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:02.003017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.002990 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:02.003206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.003185 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhw66\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-kube-api-access-lhw66\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmjhk\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:02.038191 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.038165 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:42:02.053810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.053787 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:02.081245 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:02.080340 4431 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b5bb1149aea2: secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:02.081245 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:02.080368 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr: secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:02.081245 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:02.080421 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials podName:a4568ed0-e18c-427f-9cba-ff7c960b6d6a nodeName:}" failed. No retries permitted until 2026-10-30 06:42:03.080403441 +0000 UTC m=+2931.886925183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials") pod "payload-processing-external-model-5cd7f677db-sqssr" (UID: "a4568ed0-e18c-427f-9cba-ff7c960b6d6a") : secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:02.185400 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.185364 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-plugins-config-volume\") pod \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " Oct 30 06:42:02.185710 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.185687 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-provider-credentials\") pod \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " Oct 30 06:42:02.185858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.185838 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-routing-overlay-volume\") pod \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " Oct 30 06:42:02.186004 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.185989 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-kube-api-access-dwjds\") pod \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\" (UID: \"3e5555c9-512c-4d75-9aeb-8ec2524b9496\") " Oct 30 06:42:02.197950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.190428 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-kube-api-access-dwjds" pod "3e5555c9-512c-4d75-9aeb-8ec2524b9496" (UID: "3e5555c9-512c-4d75-9aeb-8ec2524b9496"). InnerVolumeSpecName "kube-api-access-dwjds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:02.197950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.190842 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-routing-overlay-volume" pod "3e5555c9-512c-4d75-9aeb-8ec2524b9496" (UID: "3e5555c9-512c-4d75-9aeb-8ec2524b9496"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:02.197950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.191205 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-plugins-config-volume" pod "3e5555c9-512c-4d75-9aeb-8ec2524b9496" (UID: "3e5555c9-512c-4d75-9aeb-8ec2524b9496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:02.205888 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.205844 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-provider-credentials" pod "3e5555c9-512c-4d75-9aeb-8ec2524b9496" (UID: "3e5555c9-512c-4d75-9aeb-8ec2524b9496"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:02.288979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.288892 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:02.288979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.288922 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:02.288979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.288932 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e5555c9-512c-4d75-9aeb-8ec2524b9496-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:02.288979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.288940 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/3e5555c9-512c-4d75-9aeb-8ec2524b9496-kube-api-access-dwjds\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:02.402799 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.400076 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk"] Oct 30 06:42:02.406172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.406136 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:42:02.406343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.406224 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:42:02.406343 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.406270 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:42:02.455800 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.455773 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" event={"ID":"55c1bc35-ccc6-43d1-9694-494bba09ab8e","Type":"ContainerStarted","Data":"858f931c1c4230e8efb32d3a7cb5bd89162dc66d849dd18cd3e38162a5c88f8f"} Oct 30 06:42:02.457513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.457491 4431 generic.go:356] "Generic (PLEG): container finished" podID="3e5555c9-512c-4d75-9aeb-8ec2524b9496" containerID="36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe" exitCode=0 Oct 30 06:42:02.457729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.457704 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" event={"ID":"3e5555c9-512c-4d75-9aeb-8ec2524b9496","Type":"ContainerDied","Data":"36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe"} Oct 30 06:42:02.457810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.457759 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" event={"ID":"3e5555c9-512c-4d75-9aeb-8ec2524b9496","Type":"ContainerDied","Data":"96b473bf01c1efb31c8a0de768ec941e552f87bbe3e77d32cad5a79d02f2d0cb"} Oct 30 06:42:02.457810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.457783 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p" Oct 30 06:42:02.457810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.457787 4431 scope.go:122] "RemoveContainer" containerID="36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe" Oct 30 06:42:02.471126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.471104 4431 scope.go:122] "RemoveContainer" containerID="36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe" Oct 30 06:42:02.471473 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:02.471450 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe\": container with ID starting with 36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe not found: ID does not exist" containerID="36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe" Oct 30 06:42:02.471559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.471484 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe"} err="failed to get container status \"36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe\": rpc error: code = NotFound desc = could not find container \"36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe\": container with ID starting with 36c863fb33efbe8b3fd6c3d26c5d1daeb78ac5b3c74001da0f373633a86567fe not found: ID does not exist" Oct 30 06:42:02.581722 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.581600 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p"] Oct 30 06:42:02.585676 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:02.585647 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b5448f9b9-hx69p"] Oct 30 06:42:03.100647 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:03.100610 4431 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b5bb1149aea2: secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:03.100647 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:03.100644 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr: secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:03.100878 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:03.100695 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials podName:a4568ed0-e18c-427f-9cba-ff7c960b6d6a nodeName:}" failed. No retries permitted until 2026-10-30 06:42:05.100682247 +0000 UTC m=+2933.907203987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials") pod "payload-processing-external-model-5cd7f677db-sqssr" (UID: "a4568ed0-e18c-427f-9cba-ff7c960b6d6a") : secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:03.339691 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:03.339657 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5555c9-512c-4d75-9aeb-8ec2524b9496" path="/var/lib/kubelet/pods/3e5555c9-512c-4d75-9aeb-8ec2524b9496/volumes" Oct 30 06:42:03.493883 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:03.493721 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" event={"ID":"55c1bc35-ccc6-43d1-9694-494bba09ab8e","Type":"ContainerStarted","Data":"ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5"} Oct 30 06:42:03.584326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:03.581484 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" podStartSLOduration=2.581464369 podStartE2EDuration="2.581464369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:42:03.548778746 +0000 UTC m=+2932.355300495" watchObservedRunningTime="2026-10-30 06:42:03.581464369 +0000 UTC m=+2932.387986117" Oct 30 06:42:03.714259 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:03.714223 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:42:04.054034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:04.054005 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:04.055858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:04.055761 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:42:04.055858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:04.055820 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 06:42:05.054462 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:05.054426 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:42:05.054857 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:05.054481 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 06:42:05.183327 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:05.179722 4431 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b5bb1149aea2: secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:05.183327 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:05.179761 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr: secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:05.183327 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:05.179823 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials podName:a4568ed0-e18c-427f-9cba-ff7c960b6d6a nodeName:}" failed. No retries permitted until 2026-10-30 06:42:09.179803965 +0000 UTC m=+2937.986325694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials") pod "payload-processing-external-model-5cd7f677db-sqssr" (UID: "a4568ed0-e18c-427f-9cba-ff7c960b6d6a") : secret "agc-cred-b5bb1149aea2" not found Oct 30 06:42:06.054895 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.054860 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:42:06.055539 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.055506 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 06:42:06.897061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.888376 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj"] Oct 30 06:42:06.897972 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.897945 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3e5555c9-512c-4d75-9aeb-8ec2524b9496" containerName="payload-processing" Oct 30 06:42:06.898156 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.898140 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5555c9-512c-4d75-9aeb-8ec2524b9496" containerName="payload-processing" Oct 30 06:42:06.898575 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.898559 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5555c9-512c-4d75-9aeb-8ec2524b9496" containerName="payload-processing" Oct 30 06:42:06.906428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.905821 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj"] Oct 30 06:42:06.906428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.905915 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:06.932248 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932225 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"agc-cred-1868f6e16d0a\"" Oct 30 06:42:06.932387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932228 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"payload-processing-external-model-e2e--9a02ae7b-dockercfg-sbxbp\"" Oct 30 06:42:06.932387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932355 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"kube-root-ca.crt\"" Oct 30 06:42:06.932470 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932386 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"routing-overlay\"" Oct 30 06:42:06.932470 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932400 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"openshift-service-ca.crt\"" Oct 30 06:42:06.932470 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932417 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c44730\"" Oct 30 06:42:06.932694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932669 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:06.932851 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932834 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:06.932898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932866 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92sj8\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-kube-api-access-92sj8\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:06.932996 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:06.932984 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.055470 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.042785 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.056278 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.056240 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.056477 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.056458 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92sj8\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-kube-api-access-92sj8\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.073604 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.073566 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.073946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.065765 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:42:07.074132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.074105 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 06:42:07.074600 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.062343 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.081325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.078040 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.118326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.087027 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.119359 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.119275 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92sj8\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-kube-api-access-92sj8\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.226658 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.226622 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:07.458018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.457980 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" podUID="a4568ed0-e18c-427f-9cba-ff7c960b6d6a" containerName="payload-processing" containerID="cri-o://ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1" gracePeriod=25 Oct 30 06:42:07.608602 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.608572 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj"] Oct 30 06:42:07.609267 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:42:07.609232 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa537473_6ba2_4f9b_bcdb_74e290d522bd.slice/crio-fbfce4741bdebf04542aecddfcc13cf24f833f09d25f6baa3dc64851f2177ef8 WatchSource:0}: Error finding container fbfce4741bdebf04542aecddfcc13cf24f833f09d25f6baa3dc64851f2177ef8: Status 404 returned error can't find the container with id fbfce4741bdebf04542aecddfcc13cf24f833f09d25f6baa3dc64851f2177ef8 Oct 30 06:42:07.789752 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.789722 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:42:07.906179 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.906143 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-kube-api-access-p8gk8\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-kube-api-access-p8gk8\") pod \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " Oct 30 06:42:07.906395 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.906206 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-routing-overlay-volume\") pod \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " Oct 30 06:42:07.906395 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.906297 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials\") pod \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " Oct 30 06:42:07.906559 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.906493 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-plugins-config-volume\") pod \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\" (UID: \"a4568ed0-e18c-427f-9cba-ff7c960b6d6a\") " Oct 30 06:42:07.907022 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.906980 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-routing-overlay-volume" pod "a4568ed0-e18c-427f-9cba-ff7c960b6d6a" (UID: "a4568ed0-e18c-427f-9cba-ff7c960b6d6a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:07.907145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.907098 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:07.907634 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.907601 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-plugins-config-volume" pod "a4568ed0-e18c-427f-9cba-ff7c960b6d6a" (UID: "a4568ed0-e18c-427f-9cba-ff7c960b6d6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:07.909414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.909391 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-kube-api-access-p8gk8" pod "a4568ed0-e18c-427f-9cba-ff7c960b6d6a" (UID: "a4568ed0-e18c-427f-9cba-ff7c960b6d6a"). InnerVolumeSpecName "kube-api-access-p8gk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:07.909695 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:07.909677 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials" pod "a4568ed0-e18c-427f-9cba-ff7c960b6d6a" (UID: "a4568ed0-e18c-427f-9cba-ff7c960b6d6a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:08.008537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.008456 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gk8\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-kube-api-access-p8gk8\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:08.008537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.008485 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:08.008537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.008497 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4568ed0-e18c-427f-9cba-ff7c960b6d6a-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:08.055046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.055011 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:42:08.055204 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.055072 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 06:42:08.555583 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.555541 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" event={"ID":"aa537473-6ba2-4f9b-bcdb-74e290d522bd","Type":"ContainerStarted","Data":"c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16"} Oct 30 06:42:08.556094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.555604 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" event={"ID":"aa537473-6ba2-4f9b-bcdb-74e290d522bd","Type":"ContainerStarted","Data":"fbfce4741bdebf04542aecddfcc13cf24f833f09d25f6baa3dc64851f2177ef8"} Oct 30 06:42:08.556094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.555633 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:08.557172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.557143 4431 generic.go:356] "Generic (PLEG): container finished" podID="a4568ed0-e18c-427f-9cba-ff7c960b6d6a" containerID="ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1" exitCode=0 Oct 30 06:42:08.557318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.557177 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" Oct 30 06:42:08.557318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.557183 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" event={"ID":"a4568ed0-e18c-427f-9cba-ff7c960b6d6a","Type":"ContainerDied","Data":"ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1"} Oct 30 06:42:08.557318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.557234 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr" event={"ID":"a4568ed0-e18c-427f-9cba-ff7c960b6d6a","Type":"ContainerDied","Data":"bdc93ad8c215c4115a6f1bbc87948e6a9e4525d25298dac29eba6ac842f641b7"} Oct 30 06:42:08.557318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.557266 4431 scope.go:122] "RemoveContainer" containerID="ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1" Oct 30 06:42:08.608434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.608379 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" podStartSLOduration=2.608357403 podStartE2EDuration="2.608357403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:42:08.60614789 +0000 UTC m=+2937.412669639" watchObservedRunningTime="2026-10-30 06:42:08.608357403 +0000 UTC m=+2937.414879149" Oct 30 06:42:08.610906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.610884 4431 scope.go:122] "RemoveContainer" containerID="ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1" Oct 30 06:42:08.611356 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:08.611291 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1\": container with ID starting with ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1 not found: ID does not exist" containerID="ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1" Oct 30 06:42:08.611356 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.611341 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1"} err="failed to get container status \"ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1\": rpc error: code = NotFound desc = could not find container \"ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1\": container with ID starting with ce2ff0f68a18239fe4be674f1d2dffcd61583bea60948652b6b3e4d5ee58adb1 not found: ID does not exist" Oct 30 06:42:08.658550 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.658519 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr"] Oct 30 06:42:08.687235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:08.687197 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cd7f677db-sqssr"] Oct 30 06:42:09.054616 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:09.054583 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:42:09.054805 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:09.054643 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 06:42:09.339965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:09.339877 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4568ed0-e18c-427f-9cba-ff7c960b6d6a" path="/var/lib/kubelet/pods/a4568ed0-e18c-427f-9cba-ff7c960b6d6a/volumes" Oct 30 06:42:10.054636 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:10.054599 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:42:10.055029 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:10.054671 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 06:42:11.431471 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:11.431442 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:11.431978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:11.431943 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:11.433296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:11.433269 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:42:13.142401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.142363 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc"] Oct 30 06:42:13.143498 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.143472 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a4568ed0-e18c-427f-9cba-ff7c960b6d6a" containerName="payload-processing" Oct 30 06:42:13.143632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.143499 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4568ed0-e18c-427f-9cba-ff7c960b6d6a" containerName="payload-processing" Oct 30 06:42:13.143678 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.143667 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4568ed0-e18c-427f-9cba-ff7c960b6d6a" containerName="payload-processing" Oct 30 06:42:13.147980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.147957 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.164210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.164185 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc"] Oct 30 06:42:13.170731 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.170706 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c44730\"/\"agc-cred-86e1e6c7da99\"" Oct 30 06:42:13.199847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.199822 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.200001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.199927 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.200069 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.200050 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.200294 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.200273 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnw7w\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-kube-api-access-gnw7w\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.301123 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.301093 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnw7w\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-kube-api-access-gnw7w\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.301263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.301156 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.301263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.301218 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.301393 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.301283 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.301946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.301922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.302067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.302022 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.304390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.304365 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.324891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.324868 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnw7w\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-kube-api-access-gnw7w\") pod \"payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.463324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.463274 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:13.770473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.770444 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc"] Oct 30 06:42:13.771701 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:42:13.771668 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9140fb_ffaa_4ceb_bc16_1a67e0ba3731.slice/crio-5443f9f24c201066d6a0876f852412ef496664c75ed38794361435d1a44438de WatchSource:0}: Error finding container 5443f9f24c201066d6a0876f852412ef496664c75ed38794361435d1a44438de: Status 404 returned error can't find the container with id 5443f9f24c201066d6a0876f852412ef496664c75ed38794361435d1a44438de Oct 30 06:42:13.995839 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:13.994593 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6"] Oct 30 06:42:14.001034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.001009 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.012935 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.012905 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6"] Oct 30 06:42:14.017515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.017485 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"agc-cred-00345f51dea9\"" Oct 30 06:42:14.017672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.017573 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-e2e--4e48ea35-dockercfg-v22pn\"" Oct 30 06:42:14.020019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.019957 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"agc-cred-39512e62d644\"" Oct 30 06:42:14.020756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.020734 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88acce\"" Oct 30 06:42:14.020973 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.020947 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"kube-root-ca.crt\"" Oct 30 06:42:14.029517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.029453 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"openshift-service-ca.crt\"" Oct 30 06:42:14.029830 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.029647 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88acce\"/\"routing-overlay\"" Oct 30 06:42:14.112613 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.112548 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.112773 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.112658 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.112819 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.112768 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhplm\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-kube-api-access-qhplm\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.112857 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.112845 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.214388 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.214358 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.214777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.214471 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.214777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.214515 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.214777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.214595 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhplm\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-kube-api-access-qhplm\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.215267 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.215238 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.215416 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.215273 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.216990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.216973 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.237985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.237963 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhplm\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-kube-api-access-qhplm\") pod \"payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.319088 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.319022 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:14.635405 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:42:14.635372 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8479c11b_580a_4bd4_8007_4726bb0880ce.slice/crio-9d40ade1d091988577922096e5473a0d997987153bf9e25d4378f7c1dcff6ee8 WatchSource:0}: Error finding container 9d40ade1d091988577922096e5473a0d997987153bf9e25d4378f7c1dcff6ee8: Status 404 returned error can't find the container with id 9d40ade1d091988577922096e5473a0d997987153bf9e25d4378f7c1dcff6ee8 Oct 30 06:42:14.635405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.635376 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6"] Oct 30 06:42:14.653090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.653038 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" event={"ID":"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731","Type":"ContainerStarted","Data":"bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3"} Oct 30 06:42:14.653239 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.653107 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" event={"ID":"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731","Type":"ContainerStarted","Data":"5443f9f24c201066d6a0876f852412ef496664c75ed38794361435d1a44438de"} Oct 30 06:42:14.653239 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.653142 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:14.654602 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.654575 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" event={"ID":"8479c11b-580a-4bd4-8007-4726bb0880ce","Type":"ContainerStarted","Data":"9d40ade1d091988577922096e5473a0d997987153bf9e25d4378f7c1dcff6ee8"} Oct 30 06:42:14.701518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:14.701450 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" podStartSLOduration=1.701434308 podStartE2EDuration="1.701434308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:42:14.696824684 +0000 UTC m=+2943.503346433" watchObservedRunningTime="2026-10-30 06:42:14.701434308 +0000 UTC m=+2943.507956055" Oct 30 06:42:15.663771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:15.663734 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" event={"ID":"8479c11b-580a-4bd4-8007-4726bb0880ce","Type":"ContainerStarted","Data":"fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103"} Oct 30 06:42:15.664211 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:15.663803 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:15.706212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:15.706168 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" podStartSLOduration=2.70615571 podStartE2EDuration="2.70615571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:42:15.7041868 +0000 UTC m=+2944.510708548" watchObservedRunningTime="2026-10-30 06:42:15.70615571 +0000 UTC m=+2944.512677458" Oct 30 06:42:19.567877 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:19.567844 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:23.825326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:23.824521 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" podUID="80db6c09-1940-4274-a21c-57ab7dd6738c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:42:23.867423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:23.847914 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:42:23.867423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:23.847984 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:42:23.896942 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:23.896464 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:42:25.667214 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:25.667183 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:42:25.746091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:25.746055 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj"] Oct 30 06:42:26.673552 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:26.673527 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:42:30.747020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:30.746978 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" podUID="aa537473-6ba2-4f9b-bcdb-74e290d522bd" containerName="payload-processing" containerID="cri-o://c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16" gracePeriod=25 Oct 30 06:42:31.051534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.051506 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:31.112372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.112343 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-kube-api-access-92sj8\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-kube-api-access-92sj8\") pod \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " Oct 30 06:42:31.112372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.112377 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-provider-credentials\") pod \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " Oct 30 06:42:31.112694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.112655 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-plugins-config-volume\") pod \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " Oct 30 06:42:31.112795 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.112768 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-routing-overlay-volume\") pod \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\" (UID: \"aa537473-6ba2-4f9b-bcdb-74e290d522bd\") " Oct 30 06:42:31.113127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.113098 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-plugins-config-volume" pod "aa537473-6ba2-4f9b-bcdb-74e290d522bd" (UID: "aa537473-6ba2-4f9b-bcdb-74e290d522bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:31.113527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.113278 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-routing-overlay-volume" pod "aa537473-6ba2-4f9b-bcdb-74e290d522bd" (UID: "aa537473-6ba2-4f9b-bcdb-74e290d522bd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:31.113527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.113482 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:31.113527 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.113503 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa537473-6ba2-4f9b-bcdb-74e290d522bd-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:31.114762 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.114741 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-kube-api-access-92sj8" pod "aa537473-6ba2-4f9b-bcdb-74e290d522bd" (UID: "aa537473-6ba2-4f9b-bcdb-74e290d522bd"). InnerVolumeSpecName "kube-api-access-92sj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:31.114999 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.114972 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-provider-credentials" pod "aa537473-6ba2-4f9b-bcdb-74e290d522bd" (UID: "aa537473-6ba2-4f9b-bcdb-74e290d522bd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:31.214530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.214498 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92sj8\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-kube-api-access-92sj8\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:31.214530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.214527 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa537473-6ba2-4f9b-bcdb-74e290d522bd-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:42:31.874425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.874389 4431 generic.go:356] "Generic (PLEG): container finished" podID="aa537473-6ba2-4f9b-bcdb-74e290d522bd" containerID="c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16" exitCode=0 Oct 30 06:42:31.874813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.874442 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" Oct 30 06:42:31.874813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.874462 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" event={"ID":"aa537473-6ba2-4f9b-bcdb-74e290d522bd","Type":"ContainerDied","Data":"c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16"} Oct 30 06:42:31.874813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.874515 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj" event={"ID":"aa537473-6ba2-4f9b-bcdb-74e290d522bd","Type":"ContainerDied","Data":"fbfce4741bdebf04542aecddfcc13cf24f833f09d25f6baa3dc64851f2177ef8"} Oct 30 06:42:31.874813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.874537 4431 scope.go:122] "RemoveContainer" containerID="c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16" Oct 30 06:42:31.885251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.885228 4431 scope.go:122] "RemoveContainer" containerID="c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16" Oct 30 06:42:31.885540 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:31.885519 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16\": container with ID starting with c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16 not found: ID does not exist" containerID="c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16" Oct 30 06:42:31.885606 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.885547 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16"} err="failed to get container status \"c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16\": rpc error: code = NotFound desc = could not find container \"c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16\": container with ID starting with c35d19d742ed2b62deff3608d69a8e1902817b010602a29c2427d23f2b74fe16 not found: ID does not exist" Oct 30 06:42:31.926289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.926265 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj"] Oct 30 06:42:31.938280 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:31.938246 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-7c99nsshj"] Oct 30 06:42:33.341185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:33.341147 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa537473-6ba2-4f9b-bcdb-74e290d522bd" path="/var/lib/kubelet/pods/aa537473-6ba2-4f9b-bcdb-74e290d522bd/volumes" Oct 30 06:42:33.703723 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:42:33.703682 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:42:57.207513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:57.207482 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l"] Oct 30 06:42:57.208080 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:42:57.207710 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" containerID="cri-o://567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2" gracePeriod=30 Oct 30 06:43:02.575602 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.575442 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:43:02.911543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.909371 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-kube-api-access-jr6dk\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-kube-api-access-jr6dk\") pod \"1f462dfd-9c1e-451c-9a45-07a936cccb60\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " Oct 30 06:43:02.911543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.909463 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-token\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-token\") pod \"1f462dfd-9c1e-451c-9a45-07a936cccb60\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " Oct 30 06:43:02.911543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.909493 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f462dfd-9c1e-451c-9a45-07a936cccb60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f462dfd-9c1e-451c-9a45-07a936cccb60-istiod-ca-cert\") pod \"1f462dfd-9c1e-451c-9a45-07a936cccb60\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " Oct 30 06:43:02.911543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.909551 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-podinfo\") pod \"1f462dfd-9c1e-451c-9a45-07a936cccb60\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " Oct 30 06:43:02.911543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.909618 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-certs\") pod \"1f462dfd-9c1e-451c-9a45-07a936cccb60\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " Oct 30 06:43:02.911543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.909719 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-data\") pod \"1f462dfd-9c1e-451c-9a45-07a936cccb60\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " Oct 30 06:43:02.911543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.909783 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-credential-socket\") pod \"1f462dfd-9c1e-451c-9a45-07a936cccb60\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " Oct 30 06:43:02.911543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.909882 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-envoy\") pod \"1f462dfd-9c1e-451c-9a45-07a936cccb60\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " Oct 30 06:43:02.911543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.910013 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-socket\") pod \"1f462dfd-9c1e-451c-9a45-07a936cccb60\" (UID: \"1f462dfd-9c1e-451c-9a45-07a936cccb60\") " Oct 30 06:43:02.917714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.917389 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-data" pod "1f462dfd-9c1e-451c-9a45-07a936cccb60" (UID: "1f462dfd-9c1e-451c-9a45-07a936cccb60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:02.918133 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.917911 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-socket" pod "1f462dfd-9c1e-451c-9a45-07a936cccb60" (UID: "1f462dfd-9c1e-451c-9a45-07a936cccb60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:02.918591 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.918284 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-credential-socket" pod "1f462dfd-9c1e-451c-9a45-07a936cccb60" (UID: "1f462dfd-9c1e-451c-9a45-07a936cccb60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:02.919145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.918845 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f462dfd-9c1e-451c-9a45-07a936cccb60-istiod-ca-cert" pod "1f462dfd-9c1e-451c-9a45-07a936cccb60" (UID: "1f462dfd-9c1e-451c-9a45-07a936cccb60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:43:02.949335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:02.944550 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-certs" pod "1f462dfd-9c1e-451c-9a45-07a936cccb60" (UID: "1f462dfd-9c1e-451c-9a45-07a936cccb60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:03.097758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.097213 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.097758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.097254 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f462dfd-9c1e-451c-9a45-07a936cccb60-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.097758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.097270 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.097758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.097284 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.097758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.097315 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.212584 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.212534 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-kube-api-access-jr6dk" pod "1f462dfd-9c1e-451c-9a45-07a936cccb60" (UID: "1f462dfd-9c1e-451c-9a45-07a936cccb60"). InnerVolumeSpecName "kube-api-access-jr6dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:43:03.259329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.246644 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-podinfo" pod "1f462dfd-9c1e-451c-9a45-07a936cccb60" (UID: "1f462dfd-9c1e-451c-9a45-07a936cccb60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:43:03.259329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.258122 4431 generic.go:356] "Generic (PLEG): container finished" podID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerID="567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2" exitCode=0 Oct 30 06:43:03.259329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.258277 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" event={"ID":"1f462dfd-9c1e-451c-9a45-07a936cccb60","Type":"ContainerDied","Data":"567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2"} Oct 30 06:43:03.259329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.258352 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" event={"ID":"1f462dfd-9c1e-451c-9a45-07a936cccb60","Type":"ContainerDied","Data":"6e48121dfac0c41d89c2669635decd1dfa1cca6ed51a7e4f417c0d3f25c3e1ac"} Oct 30 06:43:03.259329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.258388 4431 scope.go:122] "RemoveContainer" containerID="567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2" Oct 30 06:43:03.259329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.258558 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l" Oct 30 06:43:03.279714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.279509 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-envoy" pod "1f462dfd-9c1e-451c-9a45-07a936cccb60" (UID: "1f462dfd-9c1e-451c-9a45-07a936cccb60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:43:03.318058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.317984 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.318058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.318017 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr6dk\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-kube-api-access-jr6dk\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.318058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.318033 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.378545 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.378506 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-token" pod "1f462dfd-9c1e-451c-9a45-07a936cccb60" (UID: "1f462dfd-9c1e-451c-9a45-07a936cccb60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:43:03.437221 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.437189 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f462dfd-9c1e-451c-9a45-07a936cccb60-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:43:03.454689 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.454666 4431 scope.go:122] "RemoveContainer" containerID="567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2" Oct 30 06:43:03.455293 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:43:03.455268 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2\": container with ID starting with 567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2 not found: ID does not exist" containerID="567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2" Oct 30 06:43:03.455476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.455447 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2"} err="failed to get container status \"567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2\": rpc error: code = NotFound desc = could not find container \"567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2\": container with ID starting with 567fafd37b18ce9d06befdc1809fddf3ab7a193cba05ab4c5a6809a34eeaf7f2 not found: ID does not exist" Oct 30 06:43:03.677324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.643624 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l"] Oct 30 06:43:03.707201 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.707085 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-ddc2l"] Oct 30 06:43:03.724717 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.724677 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" podUID="1d337a85-2994-4119-8d76-c0bbaca18d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:43:03.855538 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:43:03.855470 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:43:03.912424 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.911373 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 06:43:03.936883 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:03.933847 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" podUID="4199dace-66ce-4982-9ff6-629345b077ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 06:43:04.166511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:04.166418 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" podUID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:43:04.831050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:04.830394 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" podUID="80db6c09-1940-4274-a21c-57ab7dd6738c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:43:05.341212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:05.341175 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" path="/var/lib/kubelet/pods/1f462dfd-9c1e-451c-9a45-07a936cccb60/volumes" Oct 30 06:43:33.701936 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:43:33.701895 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:43:54.955265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:54.955228 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s"] Oct 30 06:43:54.955785 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:43:54.955667 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" containerID="cri-o://2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6" gracePeriod=30 Oct 30 06:44:00.235556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.235496 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:44:00.326719 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.326690 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-data\") pod \"d697083f-5df2-477f-8342-74aafc9e2343\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " Oct 30 06:44:00.326907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.326741 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-kube-api-access-879xm\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-kube-api-access-879xm\") pod \"d697083f-5df2-477f-8342-74aafc9e2343\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " Oct 30 06:44:00.326907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.326783 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-socket\") pod \"d697083f-5df2-477f-8342-74aafc9e2343\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " Oct 30 06:44:00.326907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.326831 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d697083f-5df2-477f-8342-74aafc9e2343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d697083f-5df2-477f-8342-74aafc9e2343-istiod-ca-cert\") pod \"d697083f-5df2-477f-8342-74aafc9e2343\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " Oct 30 06:44:00.327091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.326924 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d697083f-5df2-477f-8342-74aafc9e2343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d697083f-5df2-477f-8342-74aafc9e2343-istio-podinfo\") pod \"d697083f-5df2-477f-8342-74aafc9e2343\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " Oct 30 06:44:00.327091 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.327012 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-data" pod "d697083f-5df2-477f-8342-74aafc9e2343" (UID: "d697083f-5df2-477f-8342-74aafc9e2343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.327208 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.327094 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-istio-token\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-istio-token\") pod \"d697083f-5df2-477f-8342-74aafc9e2343\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " Oct 30 06:44:00.327208 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.327140 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-certs\") pod \"d697083f-5df2-477f-8342-74aafc9e2343\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " Oct 30 06:44:00.327208 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.327189 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-envoy\") pod \"d697083f-5df2-477f-8342-74aafc9e2343\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " Oct 30 06:44:00.327391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.327095 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-socket" pod "d697083f-5df2-477f-8342-74aafc9e2343" (UID: "d697083f-5df2-477f-8342-74aafc9e2343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.327391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.327276 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-credential-socket\") pod \"d697083f-5df2-477f-8342-74aafc9e2343\" (UID: \"d697083f-5df2-477f-8342-74aafc9e2343\") " Oct 30 06:44:00.327499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.327435 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-certs" pod "d697083f-5df2-477f-8342-74aafc9e2343" (UID: "d697083f-5df2-477f-8342-74aafc9e2343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.327738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.327715 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-credential-socket" pod "d697083f-5df2-477f-8342-74aafc9e2343" (UID: "d697083f-5df2-477f-8342-74aafc9e2343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.327969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.327938 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d697083f-5df2-477f-8342-74aafc9e2343-istiod-ca-cert" pod "d697083f-5df2-477f-8342-74aafc9e2343" (UID: "d697083f-5df2-477f-8342-74aafc9e2343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:00.328087 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.328067 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.328151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.328092 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.328151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.328105 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.328151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.328119 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.329475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.329437 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-kube-api-access-879xm" pod "d697083f-5df2-477f-8342-74aafc9e2343" (UID: "d697083f-5df2-477f-8342-74aafc9e2343"). InnerVolumeSpecName "kube-api-access-879xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:00.329999 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.329969 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-istio-token" pod "d697083f-5df2-477f-8342-74aafc9e2343" (UID: "d697083f-5df2-477f-8342-74aafc9e2343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:00.330250 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.330227 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d697083f-5df2-477f-8342-74aafc9e2343-istio-podinfo" pod "d697083f-5df2-477f-8342-74aafc9e2343" (UID: "d697083f-5df2-477f-8342-74aafc9e2343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:44:00.330616 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.330593 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-envoy" pod "d697083f-5df2-477f-8342-74aafc9e2343" (UID: "d697083f-5df2-477f-8342-74aafc9e2343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.431111 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.431073 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d697083f-5df2-477f-8342-74aafc9e2343-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.431111 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.431112 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.431468 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.431128 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d697083f-5df2-477f-8342-74aafc9e2343-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.431468 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.431142 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-879xm\" (UniqueName: \"kubernetes.io/projected/d697083f-5df2-477f-8342-74aafc9e2343-kube-api-access-879xm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.431468 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.431160 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d697083f-5df2-477f-8342-74aafc9e2343-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.810057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.808611 4431 generic.go:356] "Generic (PLEG): container finished" podID="d697083f-5df2-477f-8342-74aafc9e2343" containerID="2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6" exitCode=0 Oct 30 06:44:00.810057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.808785 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" event={"ID":"d697083f-5df2-477f-8342-74aafc9e2343","Type":"ContainerDied","Data":"2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6"} Oct 30 06:44:00.810057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.808835 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" event={"ID":"d697083f-5df2-477f-8342-74aafc9e2343","Type":"ContainerDied","Data":"a38a0d3dcdb5d0c0b3aa2b9b56b6c385a3b2eda9c8b89f4790da180323596fc0"} Oct 30 06:44:00.810057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.808864 4431 scope.go:122] "RemoveContainer" containerID="2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6" Oct 30 06:44:00.810057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.809018 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s" Oct 30 06:44:00.847158 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.846829 4431 scope.go:122] "RemoveContainer" containerID="2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6" Oct 30 06:44:00.850258 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:44:00.850121 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6\": container with ID starting with 2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6 not found: ID does not exist" containerID="2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6" Oct 30 06:44:00.850258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.850161 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6"} err="failed to get container status \"2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6\": rpc error: code = NotFound desc = could not find container \"2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6\": container with ID starting with 2689881fc177773c15259f7be7a480d4e05374ab77078ef1a646e9922f99e4e6 not found: ID does not exist" Oct 30 06:44:00.881435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.881409 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s"] Oct 30 06:44:00.893481 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:00.893459 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-t8p9s"] Oct 30 06:44:01.340526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:01.340495 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d697083f-5df2-477f-8342-74aafc9e2343" path="/var/lib/kubelet/pods/d697083f-5df2-477f-8342-74aafc9e2343/volumes" Oct 30 06:44:03.701642 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:44:03.701610 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:44:33.753270 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:44:33.753234 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:44:41.705323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:41.704397 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7t4cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:44:41.705323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:44:41.704512 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:45:00.185339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.184824 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq"] Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.187559 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.187586 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.187627 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.187636 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.187653 4431 cpu_manager.go:414] "containerMap: removing container" podUID="aa537473-6ba2-4f9b-bcdb-74e290d522bd" containerName="payload-processing" Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.187660 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa537473-6ba2-4f9b-bcdb-74e290d522bd" containerName="payload-processing" Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.187900 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f462dfd-9c1e-451c-9a45-07a936cccb60" containerName="istio-proxy" Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.187919 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d697083f-5df2-477f-8342-74aafc9e2343" containerName="istio-proxy" Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.187929 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa537473-6ba2-4f9b-bcdb-74e290d522bd" containerName="payload-processing" Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.192493 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n"] Oct 30 06:45:00.193367 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.192719 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:00.197529 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.197506 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v"] Oct 30 06:45:00.197734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.197699 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:00.203893 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.203871 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:00.204483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.204423 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv"] Oct 30 06:45:00.214921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.209946 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29"] Oct 30 06:45:00.214921 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.211200 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:00.220261 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.220241 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n"] Oct 30 06:45:00.220570 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.220491 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:00.228714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.228645 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq"] Oct 30 06:45:00.234022 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.233993 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v"] Oct 30 06:45:00.245334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.238158 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv"] Oct 30 06:45:00.245334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.243068 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg"] Oct 30 06:45:00.276916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.276886 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:45:00.280863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.280837 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c"] Oct 30 06:45:00.281516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.281491 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:00.287121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.287097 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:00.287238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.287006 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl"] Oct 30 06:45:00.294998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.292916 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg"] Oct 30 06:45:00.294998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.292957 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c"] Oct 30 06:45:00.294998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.292985 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn"] Oct 30 06:45:00.294998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.293521 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:00.300515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.300493 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29"] Oct 30 06:45:00.300700 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.300678 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl"] Oct 30 06:45:00.300810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.300721 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn"] Oct 30 06:45:00.300810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.300577 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:00.336719 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.336686 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e81e0bf-da55-4013-92ed-04eb35ac1572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq\" (UID: \"1e81e0bf-da55-4013-92ed-04eb35ac1572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:00.336849 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.336770 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrgh\" (UniqueName: \"kubernetes.io/projected/1e81e0bf-da55-4013-92ed-04eb35ac1572-kube-api-access-4lrgh\") pod \"maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq\" (UID: \"1e81e0bf-da55-4013-92ed-04eb35ac1572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:00.336930 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.336907 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg\" (UID: \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:00.337000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.336980 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mmc\" (UniqueName: \"kubernetes.io/projected/79acdb09-1b01-4201-a885-2d7a90b36371-kube-api-access-78mmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29\" (UID: \"79acdb09-1b01-4201-a885-2d7a90b36371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:00.337059 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.337029 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv\" (UID: \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:00.337117 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.337076 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409eb369-5875-42cf-a8d1-fec3167d20cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v\" (UID: \"409eb369-5875-42cf-a8d1-fec3167d20cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:00.337175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.337119 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7l9l\" (UniqueName: \"kubernetes.io/projected/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-kube-api-access-t7l9l\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n\" (UID: \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:00.337236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.337172 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hdw\" (UniqueName: \"kubernetes.io/projected/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-kube-api-access-97hdw\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg\" (UID: \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:00.337350 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.337333 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n\" (UID: \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:00.337423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.337390 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79acdb09-1b01-4201-a885-2d7a90b36371-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29\" (UID: \"79acdb09-1b01-4201-a885-2d7a90b36371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:00.337484 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.337447 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkjb\" (UniqueName: \"kubernetes.io/projected/409eb369-5875-42cf-a8d1-fec3167d20cd-kube-api-access-sdkjb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v\" (UID: \"409eb369-5875-42cf-a8d1-fec3167d20cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:00.337544 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.337506 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tc9r\" (UniqueName: \"kubernetes.io/projected/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-kube-api-access-8tc9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv\" (UID: \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:00.438707 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.438629 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n57dz\" (UniqueName: \"kubernetes.io/projected/463634d3-21a1-4381-a6c7-45aea8046396-kube-api-access-n57dz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c\" (UID: \"463634d3-21a1-4381-a6c7-45aea8046396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:00.438707 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.438668 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e81e0bf-da55-4013-92ed-04eb35ac1572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq\" (UID: \"1e81e0bf-da55-4013-92ed-04eb35ac1572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:00.438901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.438784 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrgh\" (UniqueName: \"kubernetes.io/projected/1e81e0bf-da55-4013-92ed-04eb35ac1572-kube-api-access-4lrgh\") pod \"maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq\" (UID: \"1e81e0bf-da55-4013-92ed-04eb35ac1572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:00.438901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.438856 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617a063-2584-41b7-8585-0f5c335fab60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn\" (UID: \"6617a063-2584-41b7-8585-0f5c335fab60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:00.438901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.438895 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg\" (UID: \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:00.439126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439039 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mmc\" (UniqueName: \"kubernetes.io/projected/79acdb09-1b01-4201-a885-2d7a90b36371-kube-api-access-78mmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29\" (UID: \"79acdb09-1b01-4201-a885-2d7a90b36371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:00.439126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439060 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv\" (UID: \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:00.439126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439085 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409eb369-5875-42cf-a8d1-fec3167d20cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v\" (UID: \"409eb369-5875-42cf-a8d1-fec3167d20cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:00.439126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439108 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7l9l\" (UniqueName: \"kubernetes.io/projected/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-kube-api-access-t7l9l\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n\" (UID: \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:00.439370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439144 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hdw\" (UniqueName: \"kubernetes.io/projected/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-kube-api-access-97hdw\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg\" (UID: \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:00.439370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439242 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n\" (UID: \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:00.439370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439260 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e81e0bf-da55-4013-92ed-04eb35ac1572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq\" (UID: \"1e81e0bf-da55-4013-92ed-04eb35ac1572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:00.439370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439274 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qlcb\" (UniqueName: \"kubernetes.io/projected/6617a063-2584-41b7-8585-0f5c335fab60-kube-api-access-4qlcb\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn\" (UID: \"6617a063-2584-41b7-8585-0f5c335fab60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:00.439593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439379 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79acdb09-1b01-4201-a885-2d7a90b36371-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29\" (UID: \"79acdb09-1b01-4201-a885-2d7a90b36371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:00.439593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439432 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkjb\" (UniqueName: \"kubernetes.io/projected/409eb369-5875-42cf-a8d1-fec3167d20cd-kube-api-access-sdkjb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v\" (UID: \"409eb369-5875-42cf-a8d1-fec3167d20cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:00.439593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439477 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tc9r\" (UniqueName: \"kubernetes.io/projected/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-kube-api-access-8tc9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv\" (UID: \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:00.439593 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439578 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl\" (UID: \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:00.439837 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439622 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463634d3-21a1-4381-a6c7-45aea8046396-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c\" (UID: \"463634d3-21a1-4381-a6c7-45aea8046396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:00.439837 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439681 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tz76\" (UniqueName: \"kubernetes.io/projected/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-kube-api-access-9tz76\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl\" (UID: \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:00.439957 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439866 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv\" (UID: \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:00.439957 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439925 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409eb369-5875-42cf-a8d1-fec3167d20cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v\" (UID: \"409eb369-5875-42cf-a8d1-fec3167d20cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:00.440065 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.439990 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg\" (UID: \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:00.440065 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.440022 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n\" (UID: \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:00.440165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.440146 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79acdb09-1b01-4201-a885-2d7a90b36371-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29\" (UID: \"79acdb09-1b01-4201-a885-2d7a90b36371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:00.462286 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.462261 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrgh\" (UniqueName: \"kubernetes.io/projected/1e81e0bf-da55-4013-92ed-04eb35ac1572-kube-api-access-4lrgh\") pod \"maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq\" (UID: \"1e81e0bf-da55-4013-92ed-04eb35ac1572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:00.462900 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.462884 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mmc\" (UniqueName: \"kubernetes.io/projected/79acdb09-1b01-4201-a885-2d7a90b36371-kube-api-access-78mmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29\" (UID: \"79acdb09-1b01-4201-a885-2d7a90b36371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:00.466891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.466863 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hdw\" (UniqueName: \"kubernetes.io/projected/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-kube-api-access-97hdw\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg\" (UID: \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:00.467016 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.466948 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkjb\" (UniqueName: \"kubernetes.io/projected/409eb369-5875-42cf-a8d1-fec3167d20cd-kube-api-access-sdkjb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v\" (UID: \"409eb369-5875-42cf-a8d1-fec3167d20cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:00.467351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.467332 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7l9l\" (UniqueName: \"kubernetes.io/projected/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-kube-api-access-t7l9l\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n\" (UID: \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:00.467453 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.467347 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tc9r\" (UniqueName: \"kubernetes.io/projected/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-kube-api-access-8tc9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv\" (UID: \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:00.541151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.541122 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tz76\" (UniqueName: \"kubernetes.io/projected/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-kube-api-access-9tz76\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl\" (UID: \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:00.541279 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.541164 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n57dz\" (UniqueName: \"kubernetes.io/projected/463634d3-21a1-4381-a6c7-45aea8046396-kube-api-access-n57dz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c\" (UID: \"463634d3-21a1-4381-a6c7-45aea8046396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:00.541279 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.541201 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617a063-2584-41b7-8585-0f5c335fab60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn\" (UID: \"6617a063-2584-41b7-8585-0f5c335fab60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:00.541416 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.541334 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qlcb\" (UniqueName: \"kubernetes.io/projected/6617a063-2584-41b7-8585-0f5c335fab60-kube-api-access-4qlcb\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn\" (UID: \"6617a063-2584-41b7-8585-0f5c335fab60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:00.541488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.541438 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl\" (UID: \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:00.541488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.541470 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463634d3-21a1-4381-a6c7-45aea8046396-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c\" (UID: \"463634d3-21a1-4381-a6c7-45aea8046396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:00.541956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.541933 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617a063-2584-41b7-8585-0f5c335fab60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn\" (UID: \"6617a063-2584-41b7-8585-0f5c335fab60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:00.542032 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.541967 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl\" (UID: \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:00.542074 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.542050 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463634d3-21a1-4381-a6c7-45aea8046396-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c\" (UID: \"463634d3-21a1-4381-a6c7-45aea8046396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:00.546544 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.546517 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:00.559212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.559187 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:00.562553 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.562525 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:00.565175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.565135 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tz76\" (UniqueName: \"kubernetes.io/projected/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-kube-api-access-9tz76\") pod \"maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl\" (UID: \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:00.565288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.565217 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qlcb\" (UniqueName: \"kubernetes.io/projected/6617a063-2584-41b7-8585-0f5c335fab60-kube-api-access-4qlcb\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn\" (UID: \"6617a063-2584-41b7-8585-0f5c335fab60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:00.565790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.565770 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n57dz\" (UniqueName: \"kubernetes.io/projected/463634d3-21a1-4381-a6c7-45aea8046396-kube-api-access-n57dz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c\" (UID: \"463634d3-21a1-4381-a6c7-45aea8046396\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:00.576457 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.576437 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:00.600465 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.600224 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:00.611963 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.611579 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:00.625039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.623569 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:00.631588 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.631563 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:00.661212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.660736 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:00.908992 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:00.908957 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq"] Oct 30 06:45:00.909580 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:00.909555 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e81e0bf_da55_4013_92ed_04eb35ac1572.slice/crio-5a6b87463ed6b47cf470e29b91654fbc1999e90a0b13dda619780d28a9b1fd94 WatchSource:0}: Error finding container 5a6b87463ed6b47cf470e29b91654fbc1999e90a0b13dda619780d28a9b1fd94: Status 404 returned error can't find the container with id 5a6b87463ed6b47cf470e29b91654fbc1999e90a0b13dda619780d28a9b1fd94 Oct 30 06:45:01.079916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.079887 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c"] Oct 30 06:45:01.083204 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:01.083180 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463634d3_21a1_4381_a6c7_45aea8046396.slice/crio-0358697a2990e0a25704a6d9e04049bb8c298d598816b3043cba8b40baefca36 WatchSource:0}: Error finding container 0358697a2990e0a25704a6d9e04049bb8c298d598816b3043cba8b40baefca36: Status 404 returned error can't find the container with id 0358697a2990e0a25704a6d9e04049bb8c298d598816b3043cba8b40baefca36 Oct 30 06:45:01.094428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.094405 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn"] Oct 30 06:45:01.098502 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.098481 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg"] Oct 30 06:45:01.098890 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:01.098863 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6617a063_2584_41b7_8585_0f5c335fab60.slice/crio-4081f12f79dda9a215607679579b5eeb54a3579ea5045e3ca5cb48e8ae7b01c4 WatchSource:0}: Error finding container 4081f12f79dda9a215607679579b5eeb54a3579ea5045e3ca5cb48e8ae7b01c4: Status 404 returned error can't find the container with id 4081f12f79dda9a215607679579b5eeb54a3579ea5045e3ca5cb48e8ae7b01c4 Oct 30 06:45:01.099622 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:01.099593 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffff3eb1_6a5f_422c_9a4e_c52892ee3577.slice/crio-3bbba47c346115df83254d6eea15c94e8ac68c2fb0818931626daf5b03ac13ff WatchSource:0}: Error finding container 3bbba47c346115df83254d6eea15c94e8ac68c2fb0818931626daf5b03ac13ff: Status 404 returned error can't find the container with id 3bbba47c346115df83254d6eea15c94e8ac68c2fb0818931626daf5b03ac13ff Oct 30 06:45:01.100922 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:01.100896 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ee227b_e549_4ab1_b046_fdcd9f6b7409.slice/crio-a14b025fb85ce84f872611517279e8a1ea315eae825d36285855b95fae3e0454 WatchSource:0}: Error finding container a14b025fb85ce84f872611517279e8a1ea315eae825d36285855b95fae3e0454: Status 404 returned error can't find the container with id a14b025fb85ce84f872611517279e8a1ea315eae825d36285855b95fae3e0454 Oct 30 06:45:01.103101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.103078 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl"] Oct 30 06:45:01.123317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.123283 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n"] Oct 30 06:45:01.125436 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:01.125413 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ec2fe4_db3f_456c_86c3_3a9ce49fba37.slice/crio-7fa581131c2619e04c65e7c85cf877861d050fa1baac7b75e15583f90fb8b784 WatchSource:0}: Error finding container 7fa581131c2619e04c65e7c85cf877861d050fa1baac7b75e15583f90fb8b784: Status 404 returned error can't find the container with id 7fa581131c2619e04c65e7c85cf877861d050fa1baac7b75e15583f90fb8b784 Oct 30 06:45:01.134789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.134762 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv"] Oct 30 06:45:01.142499 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:01.142406 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4eb09e0_cbe8_4865_9718_bd6eb4931bf6.slice/crio-1ebabaeeba3c3d047cc38cf2b81e3d251b16bc3f0721b434ea2c689a17c61a86 WatchSource:0}: Error finding container 1ebabaeeba3c3d047cc38cf2b81e3d251b16bc3f0721b434ea2c689a17c61a86: Status 404 returned error can't find the container with id 1ebabaeeba3c3d047cc38cf2b81e3d251b16bc3f0721b434ea2c689a17c61a86 Oct 30 06:45:01.160960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.160297 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v"] Oct 30 06:45:01.165000 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:01.164963 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409eb369_5875_42cf_a8d1_fec3167d20cd.slice/crio-3685cde110e4fe61ee30855fd60ac90cd85a2c0ac0406c10962b27832e84876d WatchSource:0}: Error finding container 3685cde110e4fe61ee30855fd60ac90cd85a2c0ac0406c10962b27832e84876d: Status 404 returned error can't find the container with id 3685cde110e4fe61ee30855fd60ac90cd85a2c0ac0406c10962b27832e84876d Oct 30 06:45:01.230985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.230962 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29"] Oct 30 06:45:01.349597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.349569 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" event={"ID":"79acdb09-1b01-4201-a885-2d7a90b36371","Type":"ContainerStarted","Data":"35db29173b8d83df5de52047933576b3a48cb76f3a3ed6a6d4ee0d2063fd8e72"} Oct 30 06:45:01.349774 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.349757 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" event={"ID":"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6","Type":"ContainerStarted","Data":"1ebabaeeba3c3d047cc38cf2b81e3d251b16bc3f0721b434ea2c689a17c61a86"} Oct 30 06:45:01.349888 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.349870 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" event={"ID":"ffff3eb1-6a5f-422c-9a4e-c52892ee3577","Type":"ContainerStarted","Data":"3bbba47c346115df83254d6eea15c94e8ac68c2fb0818931626daf5b03ac13ff"} Oct 30 06:45:01.350002 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.349986 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" event={"ID":"1e81e0bf-da55-4013-92ed-04eb35ac1572","Type":"ContainerStarted","Data":"f6f5cfac590022301f9244b03b3066131c52ed9be9d0ef900e8bb91e0b80970d"} Oct 30 06:45:01.350129 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.350113 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" event={"ID":"1e81e0bf-da55-4013-92ed-04eb35ac1572","Type":"ContainerStarted","Data":"5a6b87463ed6b47cf470e29b91654fbc1999e90a0b13dda619780d28a9b1fd94"} Oct 30 06:45:01.350711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.350685 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" event={"ID":"6617a063-2584-41b7-8585-0f5c335fab60","Type":"ContainerStarted","Data":"4081f12f79dda9a215607679579b5eeb54a3579ea5045e3ca5cb48e8ae7b01c4"} Oct 30 06:45:01.351803 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.351782 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" event={"ID":"463634d3-21a1-4381-a6c7-45aea8046396","Type":"ContainerStarted","Data":"0358697a2990e0a25704a6d9e04049bb8c298d598816b3043cba8b40baefca36"} Oct 30 06:45:01.352814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.352795 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" event={"ID":"409eb369-5875-42cf-a8d1-fec3167d20cd","Type":"ContainerStarted","Data":"3685cde110e4fe61ee30855fd60ac90cd85a2c0ac0406c10962b27832e84876d"} Oct 30 06:45:01.353886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.353864 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" event={"ID":"30ec2fe4-db3f-456c-86c3-3a9ce49fba37","Type":"ContainerStarted","Data":"7fa581131c2619e04c65e7c85cf877861d050fa1baac7b75e15583f90fb8b784"} Oct 30 06:45:01.354813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.354788 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" event={"ID":"a8ee227b-e549-4ab1-b046-fdcd9f6b7409","Type":"ContainerStarted","Data":"a14b025fb85ce84f872611517279e8a1ea315eae825d36285855b95fae3e0454"} Oct 30 06:45:01.389886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:01.389841 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" podStartSLOduration=1.389827092 podStartE2EDuration="1.389827092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:01.387896192 +0000 UTC m=+3110.194417942" watchObservedRunningTime="2026-10-30 06:45:01.389827092 +0000 UTC m=+3110.196348878" Oct 30 06:45:02.378690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.378652 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" event={"ID":"79acdb09-1b01-4201-a885-2d7a90b36371","Type":"ContainerStarted","Data":"e62c14bfa8c23d4cb5a9b0d6b4be40b727fd882916026c20ef70106c417bcf42"} Oct 30 06:45:02.381202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.381171 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" event={"ID":"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6","Type":"ContainerStarted","Data":"f150e364f654c385a7f79721e6002d3e7908d5f6168ea67f323ec54e4ee0c279"} Oct 30 06:45:02.383230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.383202 4431 generic.go:356] "Generic (PLEG): container finished" podID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerID="ce7c752f589a747a6cdee5dadc27fcc9d313ec9870dbc3d520d993b691a9f71f" exitCode=60 Oct 30 06:45:02.383412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.383237 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" event={"ID":"ffff3eb1-6a5f-422c-9a4e-c52892ee3577","Type":"ContainerDied","Data":"ce7c752f589a747a6cdee5dadc27fcc9d313ec9870dbc3d520d993b691a9f71f"} Oct 30 06:45:02.383650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.383628 4431 scope.go:122] "RemoveContainer" containerID="ce7c752f589a747a6cdee5dadc27fcc9d313ec9870dbc3d520d993b691a9f71f" Oct 30 06:45:02.385979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.385958 4431 generic.go:356] "Generic (PLEG): container finished" podID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerID="f6f5cfac590022301f9244b03b3066131c52ed9be9d0ef900e8bb91e0b80970d" exitCode=60 Oct 30 06:45:02.386077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.386052 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" event={"ID":"1e81e0bf-da55-4013-92ed-04eb35ac1572","Type":"ContainerDied","Data":"f6f5cfac590022301f9244b03b3066131c52ed9be9d0ef900e8bb91e0b80970d"} Oct 30 06:45:02.386324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.386291 4431 scope.go:122] "RemoveContainer" containerID="f6f5cfac590022301f9244b03b3066131c52ed9be9d0ef900e8bb91e0b80970d" Oct 30 06:45:02.388030 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.388008 4431 generic.go:356] "Generic (PLEG): container finished" podID="6617a063-2584-41b7-8585-0f5c335fab60" containerID="55b86ae3ceb96f69aeba95ff73ea930ba7c7bc64aba7c8cd3b55795e996c5350" exitCode=60 Oct 30 06:45:02.388115 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.388089 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" event={"ID":"6617a063-2584-41b7-8585-0f5c335fab60","Type":"ContainerDied","Data":"55b86ae3ceb96f69aeba95ff73ea930ba7c7bc64aba7c8cd3b55795e996c5350"} Oct 30 06:45:02.388342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.388294 4431 scope.go:122] "RemoveContainer" containerID="55b86ae3ceb96f69aeba95ff73ea930ba7c7bc64aba7c8cd3b55795e996c5350" Oct 30 06:45:02.389915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.389887 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" event={"ID":"463634d3-21a1-4381-a6c7-45aea8046396","Type":"ContainerStarted","Data":"5f5f6ced6f3aae5a460110c868dc69770bc5db4ec4067623a94fa14e46d851c5"} Oct 30 06:45:02.390100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.390083 4431 scope.go:122] "RemoveContainer" containerID="5f5f6ced6f3aae5a460110c868dc69770bc5db4ec4067623a94fa14e46d851c5" Oct 30 06:45:02.392526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.392505 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" event={"ID":"409eb369-5875-42cf-a8d1-fec3167d20cd","Type":"ContainerStarted","Data":"f274bf95cd31adfc4eba619376b0d76fe662cc774aea494e920193bce659fdd4"} Oct 30 06:45:02.395036 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.395012 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" event={"ID":"30ec2fe4-db3f-456c-86c3-3a9ce49fba37","Type":"ContainerStarted","Data":"8aeb51446d079ef7e38e3ec3f1458ad6ba810fbda773df220ca2080ee5307b39"} Oct 30 06:45:02.395351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.395332 4431 scope.go:122] "RemoveContainer" containerID="8aeb51446d079ef7e38e3ec3f1458ad6ba810fbda773df220ca2080ee5307b39" Oct 30 06:45:02.399650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.398972 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" event={"ID":"a8ee227b-e549-4ab1-b046-fdcd9f6b7409","Type":"ContainerStarted","Data":"f9c2d0f1e22761dcec8912a184489073e90f42fbcc56c35e900eec6fe44b24a8"} Oct 30 06:45:02.399650 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.399287 4431 scope.go:122] "RemoveContainer" containerID="f9c2d0f1e22761dcec8912a184489073e90f42fbcc56c35e900eec6fe44b24a8" Oct 30 06:45:02.420329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.420256 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" podStartSLOduration=2.420240894 podStartE2EDuration="2.420240894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:02.418935218 +0000 UTC m=+3111.225456961" watchObservedRunningTime="2026-10-30 06:45:02.420240894 +0000 UTC m=+3111.226762643" Oct 30 06:45:02.579137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.579083 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" podStartSLOduration=2.57906611 podStartE2EDuration="2.57906611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:02.576956953 +0000 UTC m=+3111.383478702" watchObservedRunningTime="2026-10-30 06:45:02.57906611 +0000 UTC m=+3111.385587859" Oct 30 06:45:02.656855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:02.656811 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" podStartSLOduration=2.656797718 podStartE2EDuration="2.656797718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:02.65415229 +0000 UTC m=+3111.460674044" watchObservedRunningTime="2026-10-30 06:45:02.656797718 +0000 UTC m=+3111.463319495" Oct 30 06:45:03.408392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.408353 4431 generic.go:356] "Generic (PLEG): container finished" podID="79acdb09-1b01-4201-a885-2d7a90b36371" containerID="e62c14bfa8c23d4cb5a9b0d6b4be40b727fd882916026c20ef70106c417bcf42" exitCode=60 Oct 30 06:45:03.408848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.408431 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" event={"ID":"79acdb09-1b01-4201-a885-2d7a90b36371","Type":"ContainerDied","Data":"e62c14bfa8c23d4cb5a9b0d6b4be40b727fd882916026c20ef70106c417bcf42"} Oct 30 06:45:03.408848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.408775 4431 scope.go:122] "RemoveContainer" containerID="e62c14bfa8c23d4cb5a9b0d6b4be40b727fd882916026c20ef70106c417bcf42" Oct 30 06:45:03.410491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.410467 4431 generic.go:356] "Generic (PLEG): container finished" podID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerID="f150e364f654c385a7f79721e6002d3e7908d5f6168ea67f323ec54e4ee0c279" exitCode=60 Oct 30 06:45:03.410631 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.410576 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" event={"ID":"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6","Type":"ContainerDied","Data":"f150e364f654c385a7f79721e6002d3e7908d5f6168ea67f323ec54e4ee0c279"} Oct 30 06:45:03.410793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.410777 4431 scope.go:122] "RemoveContainer" containerID="f150e364f654c385a7f79721e6002d3e7908d5f6168ea67f323ec54e4ee0c279" Oct 30 06:45:03.413033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.413010 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" event={"ID":"ffff3eb1-6a5f-422c-9a4e-c52892ee3577","Type":"ContainerStarted","Data":"2b4969604ab4b7718a8677461c7c92946233861264b42c22615bacd0d154483b"} Oct 30 06:45:03.415717 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.415688 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" event={"ID":"1e81e0bf-da55-4013-92ed-04eb35ac1572","Type":"ContainerStarted","Data":"04bb4474b85a6f25265d3f4a7498212d4229d3129c280d8a154eab7ec7ed3fa4"} Oct 30 06:45:03.418668 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.418645 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" event={"ID":"6617a063-2584-41b7-8585-0f5c335fab60","Type":"ContainerStarted","Data":"ebe83a9c140e8a7ec9631578f1b5e7926bd5aeea6e4ab77bebfa9c1f59cb6fd5"} Oct 30 06:45:03.421439 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.421418 4431 generic.go:356] "Generic (PLEG): container finished" podID="463634d3-21a1-4381-a6c7-45aea8046396" containerID="5f5f6ced6f3aae5a460110c868dc69770bc5db4ec4067623a94fa14e46d851c5" exitCode=60 Oct 30 06:45:03.421530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.421496 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" event={"ID":"463634d3-21a1-4381-a6c7-45aea8046396","Type":"ContainerDied","Data":"5f5f6ced6f3aae5a460110c868dc69770bc5db4ec4067623a94fa14e46d851c5"} Oct 30 06:45:03.421605 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.421536 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" event={"ID":"463634d3-21a1-4381-a6c7-45aea8046396","Type":"ContainerStarted","Data":"3d5f0385599bc85eb681325b49332039537bcd64421eeb4fc2a0745babe31c32"} Oct 30 06:45:03.424451 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.424413 4431 generic.go:356] "Generic (PLEG): container finished" podID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerID="f274bf95cd31adfc4eba619376b0d76fe662cc774aea494e920193bce659fdd4" exitCode=60 Oct 30 06:45:03.424562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.424487 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" event={"ID":"409eb369-5875-42cf-a8d1-fec3167d20cd","Type":"ContainerDied","Data":"f274bf95cd31adfc4eba619376b0d76fe662cc774aea494e920193bce659fdd4"} Oct 30 06:45:03.424700 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.424685 4431 scope.go:122] "RemoveContainer" containerID="f274bf95cd31adfc4eba619376b0d76fe662cc774aea494e920193bce659fdd4" Oct 30 06:45:03.426562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.426528 4431 generic.go:356] "Generic (PLEG): container finished" podID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerID="8aeb51446d079ef7e38e3ec3f1458ad6ba810fbda773df220ca2080ee5307b39" exitCode=60 Oct 30 06:45:03.426655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.426599 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" event={"ID":"30ec2fe4-db3f-456c-86c3-3a9ce49fba37","Type":"ContainerDied","Data":"8aeb51446d079ef7e38e3ec3f1458ad6ba810fbda773df220ca2080ee5307b39"} Oct 30 06:45:03.426655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.426645 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" event={"ID":"30ec2fe4-db3f-456c-86c3-3a9ce49fba37","Type":"ContainerStarted","Data":"de30813f87750757c0afc70e4189d7cd9879efc6c7bd5993ca6d4c60a23485eb"} Oct 30 06:45:03.428577 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.428553 4431 generic.go:356] "Generic (PLEG): container finished" podID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerID="f9c2d0f1e22761dcec8912a184489073e90f42fbcc56c35e900eec6fe44b24a8" exitCode=60 Oct 30 06:45:03.428688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.428625 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" event={"ID":"a8ee227b-e549-4ab1-b046-fdcd9f6b7409","Type":"ContainerDied","Data":"f9c2d0f1e22761dcec8912a184489073e90f42fbcc56c35e900eec6fe44b24a8"} Oct 30 06:45:03.428688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.428660 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" event={"ID":"a8ee227b-e549-4ab1-b046-fdcd9f6b7409","Type":"ContainerStarted","Data":"b565d05d240b77c86057369b452d0605d6bfa7d4aba41afa001c3ed17ebcfb2d"} Oct 30 06:45:03.508501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.507937 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" podStartSLOduration=3.507919077 podStartE2EDuration="3.507919077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.507077813 +0000 UTC m=+3112.313599563" watchObservedRunningTime="2026-10-30 06:45:03.507919077 +0000 UTC m=+3112.314440828" Oct 30 06:45:03.612776 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.612689 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" podStartSLOduration=3.612676379 podStartE2EDuration="3.612676379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.610959666 +0000 UTC m=+3112.417481414" watchObservedRunningTime="2026-10-30 06:45:03.612676379 +0000 UTC m=+3112.419198126" Oct 30 06:45:03.691639 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.691579 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" podStartSLOduration=3.6915660839999997 podStartE2EDuration="3.691566084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.689649994 +0000 UTC m=+3112.496171790" watchObservedRunningTime="2026-10-30 06:45:03.691566084 +0000 UTC m=+3112.498087831" Oct 30 06:45:03.702780 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:03.702749 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:45:03.733409 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.733355 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" podStartSLOduration=3.733338588 podStartE2EDuration="3.733338588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.729081016 +0000 UTC m=+3112.535602765" watchObservedRunningTime="2026-10-30 06:45:03.733338588 +0000 UTC m=+3112.539860337" Oct 30 06:45:03.774439 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:03.774393 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" podStartSLOduration=3.774380401 podStartE2EDuration="3.774380401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:03.769143867 +0000 UTC m=+3112.575665617" watchObservedRunningTime="2026-10-30 06:45:03.774380401 +0000 UTC m=+3112.580902150" Oct 30 06:45:04.442382 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.442350 4431 generic.go:356] "Generic (PLEG): container finished" podID="463634d3-21a1-4381-a6c7-45aea8046396" containerID="3d5f0385599bc85eb681325b49332039537bcd64421eeb4fc2a0745babe31c32" exitCode=60 Oct 30 06:45:04.442843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.442430 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" event={"ID":"463634d3-21a1-4381-a6c7-45aea8046396","Type":"ContainerDied","Data":"3d5f0385599bc85eb681325b49332039537bcd64421eeb4fc2a0745babe31c32"} Oct 30 06:45:04.442843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.442494 4431 scope.go:122] "RemoveContainer" containerID="5f5f6ced6f3aae5a460110c868dc69770bc5db4ec4067623a94fa14e46d851c5" Oct 30 06:45:04.442843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.442833 4431 scope.go:122] "RemoveContainer" containerID="3d5f0385599bc85eb681325b49332039537bcd64421eeb4fc2a0745babe31c32" Oct 30 06:45:04.443230 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:04.443194 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c_odh-ai-gateway-infra(463634d3-21a1-4381-a6c7-45aea8046396)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" podUID="463634d3-21a1-4381-a6c7-45aea8046396" Oct 30 06:45:04.453612 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.453580 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" event={"ID":"409eb369-5875-42cf-a8d1-fec3167d20cd","Type":"ContainerStarted","Data":"a70b024cc8bcd4e5e23d0c32eec32f942d2208ca16440bfb88d3918b7b5fea6b"} Oct 30 06:45:04.456359 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.456327 4431 generic.go:356] "Generic (PLEG): container finished" podID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerID="de30813f87750757c0afc70e4189d7cd9879efc6c7bd5993ca6d4c60a23485eb" exitCode=60 Oct 30 06:45:04.456475 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.456417 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" event={"ID":"30ec2fe4-db3f-456c-86c3-3a9ce49fba37","Type":"ContainerDied","Data":"de30813f87750757c0afc70e4189d7cd9879efc6c7bd5993ca6d4c60a23485eb"} Oct 30 06:45:04.456837 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.456819 4431 scope.go:122] "RemoveContainer" containerID="de30813f87750757c0afc70e4189d7cd9879efc6c7bd5993ca6d4c60a23485eb" Oct 30 06:45:04.457371 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:04.457343 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n_odh-ai-gateway-infra(30ec2fe4-db3f-456c-86c3-3a9ce49fba37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" Oct 30 06:45:04.458848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.458823 4431 generic.go:356] "Generic (PLEG): container finished" podID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerID="b565d05d240b77c86057369b452d0605d6bfa7d4aba41afa001c3ed17ebcfb2d" exitCode=60 Oct 30 06:45:04.458962 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.458892 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" event={"ID":"a8ee227b-e549-4ab1-b046-fdcd9f6b7409","Type":"ContainerDied","Data":"b565d05d240b77c86057369b452d0605d6bfa7d4aba41afa001c3ed17ebcfb2d"} Oct 30 06:45:04.459143 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.459125 4431 scope.go:122] "RemoveContainer" containerID="b565d05d240b77c86057369b452d0605d6bfa7d4aba41afa001c3ed17ebcfb2d" Oct 30 06:45:04.462188 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:04.459427 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl_odh-ai-gateway-infra(a8ee227b-e549-4ab1-b046-fdcd9f6b7409)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" Oct 30 06:45:04.462188 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.460994 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" event={"ID":"79acdb09-1b01-4201-a885-2d7a90b36371","Type":"ContainerStarted","Data":"aec8966a8745f2812e4217c53835d8f6eab312c85eb6454d2a5a58c1276d94ac"} Oct 30 06:45:04.463267 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.463242 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" event={"ID":"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6","Type":"ContainerStarted","Data":"072f27dfc794c85e34a50cb0686f1c656f9ee4c9413a515eed95789ab067eefd"} Oct 30 06:45:04.465118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.465098 4431 scope.go:122] "RemoveContainer" containerID="8aeb51446d079ef7e38e3ec3f1458ad6ba810fbda773df220ca2080ee5307b39" Oct 30 06:45:04.466252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.466215 4431 generic.go:356] "Generic (PLEG): container finished" podID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerID="2b4969604ab4b7718a8677461c7c92946233861264b42c22615bacd0d154483b" exitCode=60 Oct 30 06:45:04.466361 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.466280 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" event={"ID":"ffff3eb1-6a5f-422c-9a4e-c52892ee3577","Type":"ContainerDied","Data":"2b4969604ab4b7718a8677461c7c92946233861264b42c22615bacd0d154483b"} Oct 30 06:45:04.466589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.466572 4431 scope.go:122] "RemoveContainer" containerID="2b4969604ab4b7718a8677461c7c92946233861264b42c22615bacd0d154483b" Oct 30 06:45:04.466906 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:04.466870 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg_odh-ai-gateway-infra(ffff3eb1-6a5f-422c-9a4e-c52892ee3577)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" Oct 30 06:45:04.468443 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.468422 4431 generic.go:356] "Generic (PLEG): container finished" podID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerID="04bb4474b85a6f25265d3f4a7498212d4229d3129c280d8a154eab7ec7ed3fa4" exitCode=60 Oct 30 06:45:04.468537 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.468498 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" event={"ID":"1e81e0bf-da55-4013-92ed-04eb35ac1572","Type":"ContainerDied","Data":"04bb4474b85a6f25265d3f4a7498212d4229d3129c280d8a154eab7ec7ed3fa4"} Oct 30 06:45:04.468703 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.468664 4431 scope.go:122] "RemoveContainer" containerID="04bb4474b85a6f25265d3f4a7498212d4229d3129c280d8a154eab7ec7ed3fa4" Oct 30 06:45:04.468910 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:04.468883 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq_odh-ai-gateway-infra(1e81e0bf-da55-4013-92ed-04eb35ac1572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" Oct 30 06:45:04.470661 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.470639 4431 generic.go:356] "Generic (PLEG): container finished" podID="6617a063-2584-41b7-8585-0f5c335fab60" containerID="ebe83a9c140e8a7ec9631578f1b5e7926bd5aeea6e4ab77bebfa9c1f59cb6fd5" exitCode=60 Oct 30 06:45:04.470763 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.470716 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" event={"ID":"6617a063-2584-41b7-8585-0f5c335fab60","Type":"ContainerDied","Data":"ebe83a9c140e8a7ec9631578f1b5e7926bd5aeea6e4ab77bebfa9c1f59cb6fd5"} Oct 30 06:45:04.470935 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.470919 4431 scope.go:122] "RemoveContainer" containerID="ebe83a9c140e8a7ec9631578f1b5e7926bd5aeea6e4ab77bebfa9c1f59cb6fd5" Oct 30 06:45:04.471219 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:04.471199 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn_odh-ai-gateway-infra(6617a063-2584-41b7-8585-0f5c335fab60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" podUID="6617a063-2584-41b7-8585-0f5c335fab60" Oct 30 06:45:04.545598 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.545577 4431 scope.go:122] "RemoveContainer" containerID="f9c2d0f1e22761dcec8912a184489073e90f42fbcc56c35e900eec6fe44b24a8" Oct 30 06:45:04.559413 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.559378 4431 scope.go:122] "RemoveContainer" containerID="ce7c752f589a747a6cdee5dadc27fcc9d313ec9870dbc3d520d993b691a9f71f" Oct 30 06:45:04.573057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.573037 4431 scope.go:122] "RemoveContainer" containerID="f6f5cfac590022301f9244b03b3066131c52ed9be9d0ef900e8bb91e0b80970d" Oct 30 06:45:04.586039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:04.586016 4431 scope.go:122] "RemoveContainer" containerID="55b86ae3ceb96f69aeba95ff73ea930ba7c7bc64aba7c8cd3b55795e996c5350" Oct 30 06:45:05.435026 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.434997 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-00345f51dea9: secret "agc-cred-00345f51dea9" not found Oct 30 06:45:05.435026 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.435025 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6: secret "agc-cred-00345f51dea9" not found Oct 30 06:45:05.435286 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.435076 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials podName:8479c11b-580a-4bd4-8007-4726bb0880ce nodeName:}" failed. No retries permitted until 2026-10-30 06:45:05.935061726 +0000 UTC m=+3114.741583478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" (UID: "8479c11b-580a-4bd4-8007-4726bb0880ce") : secret "agc-cred-00345f51dea9" not found Oct 30 06:45:05.480110 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.480086 4431 scope.go:122] "RemoveContainer" containerID="b565d05d240b77c86057369b452d0605d6bfa7d4aba41afa001c3ed17ebcfb2d" Oct 30 06:45:05.480535 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.480414 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl_odh-ai-gateway-infra(a8ee227b-e549-4ab1-b046-fdcd9f6b7409)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" Oct 30 06:45:05.481754 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.481727 4431 generic.go:356] "Generic (PLEG): container finished" podID="79acdb09-1b01-4201-a885-2d7a90b36371" containerID="aec8966a8745f2812e4217c53835d8f6eab312c85eb6454d2a5a58c1276d94ac" exitCode=60 Oct 30 06:45:05.481890 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.481796 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" event={"ID":"79acdb09-1b01-4201-a885-2d7a90b36371","Type":"ContainerDied","Data":"aec8966a8745f2812e4217c53835d8f6eab312c85eb6454d2a5a58c1276d94ac"} Oct 30 06:45:05.481890 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.481860 4431 scope.go:122] "RemoveContainer" containerID="e62c14bfa8c23d4cb5a9b0d6b4be40b727fd882916026c20ef70106c417bcf42" Oct 30 06:45:05.482071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.482050 4431 scope.go:122] "RemoveContainer" containerID="aec8966a8745f2812e4217c53835d8f6eab312c85eb6454d2a5a58c1276d94ac" Oct 30 06:45:05.482417 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.482397 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29_odh-ai-gateway-infra(79acdb09-1b01-4201-a885-2d7a90b36371)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" Oct 30 06:45:05.483870 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.483847 4431 generic.go:356] "Generic (PLEG): container finished" podID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerID="072f27dfc794c85e34a50cb0686f1c656f9ee4c9413a515eed95789ab067eefd" exitCode=60 Oct 30 06:45:05.483957 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.483923 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" event={"ID":"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6","Type":"ContainerDied","Data":"072f27dfc794c85e34a50cb0686f1c656f9ee4c9413a515eed95789ab067eefd"} Oct 30 06:45:05.484149 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.484135 4431 scope.go:122] "RemoveContainer" containerID="072f27dfc794c85e34a50cb0686f1c656f9ee4c9413a515eed95789ab067eefd" Oct 30 06:45:05.484441 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.484423 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv_odh-ai-gateway-infra(c4eb09e0-cbe8-4865-9718-bd6eb4931bf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" Oct 30 06:45:05.485831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.485815 4431 scope.go:122] "RemoveContainer" containerID="2b4969604ab4b7718a8677461c7c92946233861264b42c22615bacd0d154483b" Oct 30 06:45:05.486078 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.486058 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg_odh-ai-gateway-infra(ffff3eb1-6a5f-422c-9a4e-c52892ee3577)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" Oct 30 06:45:05.487494 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.487475 4431 scope.go:122] "RemoveContainer" containerID="04bb4474b85a6f25265d3f4a7498212d4229d3129c280d8a154eab7ec7ed3fa4" Oct 30 06:45:05.487775 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.487752 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq_odh-ai-gateway-infra(1e81e0bf-da55-4013-92ed-04eb35ac1572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" Oct 30 06:45:05.489195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.489179 4431 scope.go:122] "RemoveContainer" containerID="ebe83a9c140e8a7ec9631578f1b5e7926bd5aeea6e4ab77bebfa9c1f59cb6fd5" Oct 30 06:45:05.489443 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.489415 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn_odh-ai-gateway-infra(6617a063-2584-41b7-8585-0f5c335fab60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" podUID="6617a063-2584-41b7-8585-0f5c335fab60" Oct 30 06:45:05.490859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.490842 4431 scope.go:122] "RemoveContainer" containerID="3d5f0385599bc85eb681325b49332039537bcd64421eeb4fc2a0745babe31c32" Oct 30 06:45:05.491094 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.491074 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c_odh-ai-gateway-infra(463634d3-21a1-4381-a6c7-45aea8046396)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" podUID="463634d3-21a1-4381-a6c7-45aea8046396" Oct 30 06:45:05.492736 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.492714 4431 generic.go:356] "Generic (PLEG): container finished" podID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerID="a70b024cc8bcd4e5e23d0c32eec32f942d2208ca16440bfb88d3918b7b5fea6b" exitCode=60 Oct 30 06:45:05.492829 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.492780 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" event={"ID":"409eb369-5875-42cf-a8d1-fec3167d20cd","Type":"ContainerDied","Data":"a70b024cc8bcd4e5e23d0c32eec32f942d2208ca16440bfb88d3918b7b5fea6b"} Oct 30 06:45:05.492998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.492981 4431 scope.go:122] "RemoveContainer" containerID="a70b024cc8bcd4e5e23d0c32eec32f942d2208ca16440bfb88d3918b7b5fea6b" Oct 30 06:45:05.493268 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.493243 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v_odh-ai-gateway-infra(409eb369-5875-42cf-a8d1-fec3167d20cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" Oct 30 06:45:05.494853 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.494838 4431 scope.go:122] "RemoveContainer" containerID="de30813f87750757c0afc70e4189d7cd9879efc6c7bd5993ca6d4c60a23485eb" Oct 30 06:45:05.495037 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.495021 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n_odh-ai-gateway-infra(30ec2fe4-db3f-456c-86c3-3a9ce49fba37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" Oct 30 06:45:05.496369 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.496350 4431 scope.go:122] "RemoveContainer" containerID="f150e364f654c385a7f79721e6002d3e7908d5f6168ea67f323ec54e4ee0c279" Oct 30 06:45:05.509597 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:05.509580 4431 scope.go:122] "RemoveContainer" containerID="f274bf95cd31adfc4eba619376b0d76fe662cc774aea494e920193bce659fdd4" Oct 30 06:45:05.947721 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.947692 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-00345f51dea9: secret "agc-cred-00345f51dea9" not found Oct 30 06:45:05.947721 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.947719 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6: secret "agc-cred-00345f51dea9" not found Oct 30 06:45:05.947949 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:05.947773 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials podName:8479c11b-580a-4bd4-8007-4726bb0880ce nodeName:}" failed. No retries permitted until 2026-10-30 06:45:06.947759532 +0000 UTC m=+3115.754281259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" (UID: "8479c11b-580a-4bd4-8007-4726bb0880ce") : secret "agc-cred-00345f51dea9" not found Oct 30 06:45:06.505194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:06.505163 4431 scope.go:122] "RemoveContainer" containerID="a70b024cc8bcd4e5e23d0c32eec32f942d2208ca16440bfb88d3918b7b5fea6b" Oct 30 06:45:06.505673 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:06.505583 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v_odh-ai-gateway-infra(409eb369-5875-42cf-a8d1-fec3167d20cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" Oct 30 06:45:06.506891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:06.506866 4431 scope.go:122] "RemoveContainer" containerID="aec8966a8745f2812e4217c53835d8f6eab312c85eb6454d2a5a58c1276d94ac" Oct 30 06:45:06.507217 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:06.507192 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29_odh-ai-gateway-infra(79acdb09-1b01-4201-a885-2d7a90b36371)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" Oct 30 06:45:06.508620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:06.508601 4431 scope.go:122] "RemoveContainer" containerID="072f27dfc794c85e34a50cb0686f1c656f9ee4c9413a515eed95789ab067eefd" Oct 30 06:45:06.508921 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:06.508903 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv_odh-ai-gateway-infra(c4eb09e0-cbe8-4865-9718-bd6eb4931bf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" Oct 30 06:45:06.968886 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:06.968854 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-00345f51dea9: secret "agc-cred-00345f51dea9" not found Oct 30 06:45:06.969102 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:06.969091 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6: secret "agc-cred-00345f51dea9" not found Oct 30 06:45:06.969272 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:06.969261 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials podName:8479c11b-580a-4bd4-8007-4726bb0880ce nodeName:}" failed. No retries permitted until 2026-10-30 06:45:08.969241202 +0000 UTC m=+3117.775762943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" (UID: "8479c11b-580a-4bd4-8007-4726bb0880ce") : secret "agc-cred-00345f51dea9" not found Oct 30 06:45:07.606745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:07.606703 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6"] Oct 30 06:45:08.584291 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:08.584256 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc"] Oct 30 06:45:09.049325 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:09.048475 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-00345f51dea9: secret "agc-cred-00345f51dea9" not found Oct 30 06:45:09.049325 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:09.048505 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88acce/agc-cred-39512e62d644: secret "agc-cred-39512e62d644" not found Oct 30 06:45:09.049325 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:09.048519 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6: [secret "agc-cred-00345f51dea9" not found, secret "agc-cred-39512e62d644" not found] Oct 30 06:45:09.049325 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:09.048593 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials podName:8479c11b-580a-4bd4-8007-4726bb0880ce nodeName:}" failed. No retries permitted until 2026-10-30 06:45:13.048572885 +0000 UTC m=+3121.855094634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" (UID: "8479c11b-580a-4bd4-8007-4726bb0880ce") : [secret "agc-cred-00345f51dea9" not found, secret "agc-cred-39512e62d644" not found] Oct 30 06:45:12.607132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.607094 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" podUID="8479c11b-580a-4bd4-8007-4726bb0880ce" containerName="payload-processing" containerID="cri-o://fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103" gracePeriod=25 Oct 30 06:45:12.889197 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.889172 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:45:12.909920 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.909894 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-plugins-config-volume\") pod \"8479c11b-580a-4bd4-8007-4726bb0880ce\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " Oct 30 06:45:12.910203 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.910182 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials\") pod \"8479c11b-580a-4bd4-8007-4726bb0880ce\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " Oct 30 06:45:12.910251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.910239 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-routing-overlay-volume\") pod \"8479c11b-580a-4bd4-8007-4726bb0880ce\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " Oct 30 06:45:12.910341 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.910322 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-plugins-config-volume" pod "8479c11b-580a-4bd4-8007-4726bb0880ce" (UID: "8479c11b-580a-4bd4-8007-4726bb0880ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:12.910383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.910330 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-kube-api-access-qhplm\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-kube-api-access-qhplm\") pod \"8479c11b-580a-4bd4-8007-4726bb0880ce\" (UID: \"8479c11b-580a-4bd4-8007-4726bb0880ce\") " Oct 30 06:45:12.910671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.910646 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-routing-overlay-volume" pod "8479c11b-580a-4bd4-8007-4726bb0880ce" (UID: "8479c11b-580a-4bd4-8007-4726bb0880ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:12.911088 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.911067 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:12.911162 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.911089 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8479c11b-580a-4bd4-8007-4726bb0880ce-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:12.912360 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.912335 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials" pod "8479c11b-580a-4bd4-8007-4726bb0880ce" (UID: "8479c11b-580a-4bd4-8007-4726bb0880ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:12.912453 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:12.912384 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-kube-api-access-qhplm" pod "8479c11b-580a-4bd4-8007-4726bb0880ce" (UID: "8479c11b-580a-4bd4-8007-4726bb0880ce"). InnerVolumeSpecName "kube-api-access-qhplm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:13.012789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.012756 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:13.012789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.012781 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhplm\" (UniqueName: \"kubernetes.io/projected/8479c11b-580a-4bd4-8007-4726bb0880ce-kube-api-access-qhplm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:13.578164 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.578128 4431 generic.go:356] "Generic (PLEG): container finished" podID="8479c11b-580a-4bd4-8007-4726bb0880ce" containerID="fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103" exitCode=0 Oct 30 06:45:13.578352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.578197 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" Oct 30 06:45:13.578352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.578201 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" event={"ID":"8479c11b-580a-4bd4-8007-4726bb0880ce","Type":"ContainerDied","Data":"fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103"} Oct 30 06:45:13.578352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.578253 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6" event={"ID":"8479c11b-580a-4bd4-8007-4726bb0880ce","Type":"ContainerDied","Data":"9d40ade1d091988577922096e5473a0d997987153bf9e25d4378f7c1dcff6ee8"} Oct 30 06:45:13.578352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.578276 4431 scope.go:122] "RemoveContainer" containerID="fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103" Oct 30 06:45:13.585082 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.585056 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" podUID="5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" containerName="payload-processing" containerID="cri-o://bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3" gracePeriod=25 Oct 30 06:45:13.588115 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.588097 4431 scope.go:122] "RemoveContainer" containerID="fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103" Oct 30 06:45:13.588415 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:13.588394 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103\": container with ID starting with fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103 not found: ID does not exist" containerID="fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103" Oct 30 06:45:13.588481 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.588425 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103"} err="failed to get container status \"fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103\": rpc error: code = NotFound desc = could not find container \"fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103\": container with ID starting with fc52ac06868519707d2c34dd1a407a6f69ecbf77dce7b2de6426fdce5701f103 not found: ID does not exist" Oct 30 06:45:13.632702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.632681 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6"] Oct 30 06:45:13.638556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.638537 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88acce/payload-processing-external-model-e2e-extmod-a-88acce-855cz4kl6"] Oct 30 06:45:13.871586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.871531 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:45:13.923868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.923843 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-provider-credentials\") pod \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " Oct 30 06:45:13.924037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.923927 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-kube-api-access-gnw7w\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-kube-api-access-gnw7w\") pod \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " Oct 30 06:45:13.924037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.924031 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-plugins-config-volume\") pod \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " Oct 30 06:45:13.924148 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.924065 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-routing-overlay-volume\") pod \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\" (UID: \"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731\") " Oct 30 06:45:13.924636 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.924608 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-plugins-config-volume" pod "5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" (UID: "5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:13.924749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.924652 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-routing-overlay-volume" pod "5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" (UID: "5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:13.924749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.924731 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:13.926356 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.926325 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-provider-credentials" pod "5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" (UID: "5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:13.926718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:13.926689 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-kube-api-access-gnw7w" pod "5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" (UID: "5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731"). InnerVolumeSpecName "kube-api-access-gnw7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:14.026350 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.026325 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:14.026350 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.026349 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:14.026506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.026358 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnw7w\" (UniqueName: \"kubernetes.io/projected/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731-kube-api-access-gnw7w\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:14.330477 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.330448 4431 scope.go:122] "RemoveContainer" containerID="b565d05d240b77c86057369b452d0605d6bfa7d4aba41afa001c3ed17ebcfb2d" Oct 30 06:45:14.330621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.330550 4431 scope.go:122] "RemoveContainer" containerID="3d5f0385599bc85eb681325b49332039537bcd64421eeb4fc2a0745babe31c32" Oct 30 06:45:14.590417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.590335 4431 generic.go:356] "Generic (PLEG): container finished" podID="5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" containerID="bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3" exitCode=0 Oct 30 06:45:14.590417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.590353 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" event={"ID":"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731","Type":"ContainerDied","Data":"bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3"} Oct 30 06:45:14.590417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.590388 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" Oct 30 06:45:14.590417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.590417 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc" event={"ID":"5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731","Type":"ContainerDied","Data":"5443f9f24c201066d6a0876f852412ef496664c75ed38794361435d1a44438de"} Oct 30 06:45:14.590706 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.590446 4431 scope.go:122] "RemoveContainer" containerID="bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3" Oct 30 06:45:14.601479 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.601450 4431 scope.go:122] "RemoveContainer" containerID="bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3" Oct 30 06:45:14.601776 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:14.601753 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3\": container with ID starting with bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3 not found: ID does not exist" containerID="bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3" Oct 30 06:45:14.601886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.601786 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3"} err="failed to get container status \"bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3\": rpc error: code = NotFound desc = could not find container \"bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3\": container with ID starting with bc18635e3afe168bacd24847206b3abf913c0554592e7d23f2730c85e832d7f3 not found: ID does not exist" Oct 30 06:45:14.649584 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.649553 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc"] Oct 30 06:45:14.661311 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:14.661279 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c44730/payload-processing-external-model-e2e-extmod-b-c44730-78c7nqnhc"] Oct 30 06:45:15.331024 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.330986 4431 scope.go:122] "RemoveContainer" containerID="2b4969604ab4b7718a8677461c7c92946233861264b42c22615bacd0d154483b" Oct 30 06:45:15.331221 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.331141 4431 scope.go:122] "RemoveContainer" containerID="aec8966a8745f2812e4217c53835d8f6eab312c85eb6454d2a5a58c1276d94ac" Oct 30 06:45:15.339770 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.339747 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" path="/var/lib/kubelet/pods/5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731/volumes" Oct 30 06:45:15.340157 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.340145 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8479c11b-580a-4bd4-8007-4726bb0880ce" path="/var/lib/kubelet/pods/8479c11b-580a-4bd4-8007-4726bb0880ce/volumes" Oct 30 06:45:15.504185 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:15.504152 4431 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ee227b_e549_4ab1_b046_fdcd9f6b7409.slice/crio-conmon-1ab89cbf7d3dc4ebf813def4fec6122a6d1c6ea9d9ed819b06043ab7aa909672.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:45:15.599905 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.599877 4431 generic.go:356] "Generic (PLEG): container finished" podID="463634d3-21a1-4381-a6c7-45aea8046396" containerID="5c8de1787ed094b505ce68a035a658cbd8a5ffbb9a1ea1821f82cb9e63c6ec44" exitCode=60 Oct 30 06:45:15.600020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.599951 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" event={"ID":"463634d3-21a1-4381-a6c7-45aea8046396","Type":"ContainerDied","Data":"5c8de1787ed094b505ce68a035a658cbd8a5ffbb9a1ea1821f82cb9e63c6ec44"} Oct 30 06:45:15.600020 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.600014 4431 scope.go:122] "RemoveContainer" containerID="3d5f0385599bc85eb681325b49332039537bcd64421eeb4fc2a0745babe31c32" Oct 30 06:45:15.600481 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.600461 4431 scope.go:122] "RemoveContainer" containerID="5c8de1787ed094b505ce68a035a658cbd8a5ffbb9a1ea1821f82cb9e63c6ec44" Oct 30 06:45:15.600881 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:15.600856 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c_odh-ai-gateway-infra(463634d3-21a1-4381-a6c7-45aea8046396)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" podUID="463634d3-21a1-4381-a6c7-45aea8046396" Oct 30 06:45:15.602032 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.602005 4431 generic.go:356] "Generic (PLEG): container finished" podID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerID="1ab89cbf7d3dc4ebf813def4fec6122a6d1c6ea9d9ed819b06043ab7aa909672" exitCode=60 Oct 30 06:45:15.602100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.602077 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" event={"ID":"a8ee227b-e549-4ab1-b046-fdcd9f6b7409","Type":"ContainerDied","Data":"1ab89cbf7d3dc4ebf813def4fec6122a6d1c6ea9d9ed819b06043ab7aa909672"} Oct 30 06:45:15.602488 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.602462 4431 scope.go:122] "RemoveContainer" containerID="1ab89cbf7d3dc4ebf813def4fec6122a6d1c6ea9d9ed819b06043ab7aa909672" Oct 30 06:45:15.602799 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:15.602779 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl_odh-ai-gateway-infra(a8ee227b-e549-4ab1-b046-fdcd9f6b7409)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" Oct 30 06:45:15.611919 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:15.611903 4431 scope.go:122] "RemoveContainer" containerID="b565d05d240b77c86057369b452d0605d6bfa7d4aba41afa001c3ed17ebcfb2d" Oct 30 06:45:16.049208 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.049176 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr"] Oct 30 06:45:16.050275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.050252 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" containerName="payload-processing" Oct 30 06:45:16.050275 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.050273 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" containerName="payload-processing" Oct 30 06:45:16.050467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.050319 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8479c11b-580a-4bd4-8007-4726bb0880ce" containerName="payload-processing" Oct 30 06:45:16.050467 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.050327 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8479c11b-580a-4bd4-8007-4726bb0880ce" containerName="payload-processing" Oct 30 06:45:16.050585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.050497 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9140fb-ffaa-4ceb-bc16-1a67e0ba3731" containerName="payload-processing" Oct 30 06:45:16.050585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.050513 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8479c11b-580a-4bd4-8007-4726bb0880ce" containerName="payload-processing" Oct 30 06:45:16.054919 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.054899 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.077242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.077216 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr"] Oct 30 06:45:16.081725 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.081700 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1c148a-openshift-default-dockercfg-4drzw\"" Oct 30 06:45:16.154459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.154429 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-credential-socket\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.154583 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.154474 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-data\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.154583 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.154560 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-socket\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.154735 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.154712 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istiod-ca-cert\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.154822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.154799 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-podinfo\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.154875 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.154862 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8qq\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-kube-api-access-lt8qq\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.154915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.154896 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-certs\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.154969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.154956 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-token\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.155017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.155006 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-envoy\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.255848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.255807 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-envoy\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.255942 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-credential-socket\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.255971 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-data\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256005 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-socket\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256090 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istiod-ca-cert\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256128 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-podinfo\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256169 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256154 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8qq\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-kube-api-access-lt8qq\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256185 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-certs\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256215 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-token\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256333 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-credential-socket\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256390 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-data\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256446 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-socket\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256627 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256605 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-certs\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.256844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.256826 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istiod-ca-cert\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.258041 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.258024 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-envoy\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.258877 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.258853 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-podinfo\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.317324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.314656 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-token\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.326349 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.324410 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8qq\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-kube-api-access-lt8qq\") pod \"e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.351327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.344645 4431 scope.go:122] "RemoveContainer" containerID="072f27dfc794c85e34a50cb0686f1c656f9ee4c9413a515eed95789ab067eefd" Oct 30 06:45:16.351327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.346049 4431 scope.go:122] "RemoveContainer" containerID="de30813f87750757c0afc70e4189d7cd9879efc6c7bd5993ca6d4c60a23485eb" Oct 30 06:45:16.353661 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.352749 4431 scope.go:122] "RemoveContainer" containerID="ebe83a9c140e8a7ec9631578f1b5e7926bd5aeea6e4ab77bebfa9c1f59cb6fd5" Oct 30 06:45:16.396452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.395203 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:16.627338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.619733 4431 generic.go:356] "Generic (PLEG): container finished" podID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerID="a5538f5fd296d4cb8c06665fd5a3c143be808a421530b162252a386b97124336" exitCode=60 Oct 30 06:45:16.627338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.619801 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" event={"ID":"ffff3eb1-6a5f-422c-9a4e-c52892ee3577","Type":"ContainerDied","Data":"a5538f5fd296d4cb8c06665fd5a3c143be808a421530b162252a386b97124336"} Oct 30 06:45:16.627338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.619872 4431 scope.go:122] "RemoveContainer" containerID="2b4969604ab4b7718a8677461c7c92946233861264b42c22615bacd0d154483b" Oct 30 06:45:16.627338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.620324 4431 scope.go:122] "RemoveContainer" containerID="a5538f5fd296d4cb8c06665fd5a3c143be808a421530b162252a386b97124336" Oct 30 06:45:16.627338 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:16.620681 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg_odh-ai-gateway-infra(ffff3eb1-6a5f-422c-9a4e-c52892ee3577)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" Oct 30 06:45:16.627338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.626620 4431 generic.go:356] "Generic (PLEG): container finished" podID="79acdb09-1b01-4201-a885-2d7a90b36371" containerID="9fdee7f790e16b19e184c5a2a83fa633a5632bcfdb8092947fb1d8856c18b297" exitCode=60 Oct 30 06:45:16.627338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.626698 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" event={"ID":"79acdb09-1b01-4201-a885-2d7a90b36371","Type":"ContainerDied","Data":"9fdee7f790e16b19e184c5a2a83fa633a5632bcfdb8092947fb1d8856c18b297"} Oct 30 06:45:16.627338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.627226 4431 scope.go:122] "RemoveContainer" containerID="9fdee7f790e16b19e184c5a2a83fa633a5632bcfdb8092947fb1d8856c18b297" Oct 30 06:45:16.627798 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:16.627711 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29_odh-ai-gateway-infra(79acdb09-1b01-4201-a885-2d7a90b36371)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" Oct 30 06:45:16.666344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.656833 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c"] Oct 30 06:45:16.666344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.657702 4431 scope.go:122] "RemoveContainer" containerID="aec8966a8745f2812e4217c53835d8f6eab312c85eb6454d2a5a58c1276d94ac" Oct 30 06:45:16.695793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.695765 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl"] Oct 30 06:45:16.851826 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.851791 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr"] Oct 30 06:45:16.862804 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.862774 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:45:16.862911 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.862851 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:45:16.862911 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.862892 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:45:16.979159 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.979135 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:16.995487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:16.995466 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:17.090869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.089061 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-kube-api-access-9tz76\" (UniqueName: \"kubernetes.io/projected/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-kube-api-access-9tz76\") pod \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\" (UID: \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\") " Oct 30 06:45:17.090869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.089126 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463634d3-21a1-4381-a6c7-45aea8046396-kube-api-access-n57dz\" (UniqueName: \"kubernetes.io/projected/463634d3-21a1-4381-a6c7-45aea8046396-kube-api-access-n57dz\") pod \"463634d3-21a1-4381-a6c7-45aea8046396\" (UID: \"463634d3-21a1-4381-a6c7-45aea8046396\") " Oct 30 06:45:17.090869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.089323 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463634d3-21a1-4381-a6c7-45aea8046396-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463634d3-21a1-4381-a6c7-45aea8046396-maas-api-service-ca\") pod \"463634d3-21a1-4381-a6c7-45aea8046396\" (UID: \"463634d3-21a1-4381-a6c7-45aea8046396\") " Oct 30 06:45:17.090869 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.089643 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-maas-api-service-ca\") pod \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\" (UID: \"a8ee227b-e549-4ab1-b046-fdcd9f6b7409\") " Oct 30 06:45:17.092757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.091863 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-kube-api-access-9tz76" pod "a8ee227b-e549-4ab1-b046-fdcd9f6b7409" (UID: "a8ee227b-e549-4ab1-b046-fdcd9f6b7409"). InnerVolumeSpecName "kube-api-access-9tz76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:17.092757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.092506 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-maas-api-service-ca" pod "a8ee227b-e549-4ab1-b046-fdcd9f6b7409" (UID: "a8ee227b-e549-4ab1-b046-fdcd9f6b7409"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:17.093614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.093451 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463634d3-21a1-4381-a6c7-45aea8046396-kube-api-access-n57dz" pod "463634d3-21a1-4381-a6c7-45aea8046396" (UID: "463634d3-21a1-4381-a6c7-45aea8046396"). InnerVolumeSpecName "kube-api-access-n57dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:17.093958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.093882 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463634d3-21a1-4381-a6c7-45aea8046396-maas-api-service-ca" pod "463634d3-21a1-4381-a6c7-45aea8046396" (UID: "463634d3-21a1-4381-a6c7-45aea8046396"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:17.191975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.191939 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463634d3-21a1-4381-a6c7-45aea8046396-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:17.191975 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.191975 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:17.192154 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.191991 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tz76\" (UniqueName: \"kubernetes.io/projected/a8ee227b-e549-4ab1-b046-fdcd9f6b7409-kube-api-access-9tz76\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:17.192154 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.192005 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n57dz\" (UniqueName: \"kubernetes.io/projected/463634d3-21a1-4381-a6c7-45aea8046396-kube-api-access-n57dz\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:17.331424 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.331354 4431 scope.go:122] "RemoveContainer" containerID="04bb4474b85a6f25265d3f4a7498212d4229d3129c280d8a154eab7ec7ed3fa4" Oct 30 06:45:17.635538 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.635502 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" event={"ID":"9773d54c-d0fa-441b-b9b7-fe731d630d3a","Type":"ContainerStarted","Data":"8da47eee2ac4486c631fef1c07b0dd1c0aac445035677a776160a8db06417b29"} Oct 30 06:45:17.635677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.635571 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" event={"ID":"9773d54c-d0fa-441b-b9b7-fe731d630d3a","Type":"ContainerStarted","Data":"68b5cb3f3a505568b723ebf30e3e3c3383defb4adc5bd41b719ecc6efa8b02ae"} Oct 30 06:45:17.636974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.636954 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" Oct 30 06:45:17.636974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.636963 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c" event={"ID":"463634d3-21a1-4381-a6c7-45aea8046396","Type":"ContainerDied","Data":"0358697a2990e0a25704a6d9e04049bb8c298d598816b3043cba8b40baefca36"} Oct 30 06:45:17.637137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.637017 4431 scope.go:122] "RemoveContainer" containerID="5c8de1787ed094b505ce68a035a658cbd8a5ffbb9a1ea1821f82cb9e63c6ec44" Oct 30 06:45:17.639511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.639490 4431 generic.go:356] "Generic (PLEG): container finished" podID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerID="d9346eee1007506cfdbe096bc497d9096908244d69b8a393bea920d532bed21e" exitCode=60 Oct 30 06:45:17.639639 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.639610 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" event={"ID":"30ec2fe4-db3f-456c-86c3-3a9ce49fba37","Type":"ContainerDied","Data":"d9346eee1007506cfdbe096bc497d9096908244d69b8a393bea920d532bed21e"} Oct 30 06:45:17.639969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.639952 4431 scope.go:122] "RemoveContainer" containerID="d9346eee1007506cfdbe096bc497d9096908244d69b8a393bea920d532bed21e" Oct 30 06:45:17.640209 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:17.640188 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n_odh-ai-gateway-infra(30ec2fe4-db3f-456c-86c3-3a9ce49fba37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" Oct 30 06:45:17.641900 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.641879 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" Oct 30 06:45:17.641998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.641907 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl" event={"ID":"a8ee227b-e549-4ab1-b046-fdcd9f6b7409","Type":"ContainerDied","Data":"a14b025fb85ce84f872611517279e8a1ea315eae825d36285855b95fae3e0454"} Oct 30 06:45:17.646335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.646278 4431 generic.go:356] "Generic (PLEG): container finished" podID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerID="19e8fff7729d6f4a400cb17651e7b2a6bbca8c26d7572c5bb5aebe26308bd758" exitCode=60 Oct 30 06:45:17.646422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.646333 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" event={"ID":"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6","Type":"ContainerDied","Data":"19e8fff7729d6f4a400cb17651e7b2a6bbca8c26d7572c5bb5aebe26308bd758"} Oct 30 06:45:17.646746 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.646726 4431 scope.go:122] "RemoveContainer" containerID="19e8fff7729d6f4a400cb17651e7b2a6bbca8c26d7572c5bb5aebe26308bd758" Oct 30 06:45:17.647002 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:17.646981 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv_odh-ai-gateway-infra(c4eb09e0-cbe8-4865-9718-bd6eb4931bf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" Oct 30 06:45:17.648833 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.648817 4431 scope.go:122] "RemoveContainer" containerID="de30813f87750757c0afc70e4189d7cd9879efc6c7bd5993ca6d4c60a23485eb" Oct 30 06:45:17.650222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.650197 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" event={"ID":"6617a063-2584-41b7-8585-0f5c335fab60","Type":"ContainerStarted","Data":"985e3e417059c905f168c48e55baaa50d251c7d6b2534e03135b05cbec5a76ae"} Oct 30 06:45:17.662584 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.662554 4431 scope.go:122] "RemoveContainer" containerID="1ab89cbf7d3dc4ebf813def4fec6122a6d1c6ea9d9ed819b06043ab7aa909672" Oct 30 06:45:17.680557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.680514 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" podStartSLOduration=1.680501439 podStartE2EDuration="1.680501439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:17.679979677 +0000 UTC m=+3126.486501439" watchObservedRunningTime="2026-10-30 06:45:17.680501439 +0000 UTC m=+3126.487023186" Oct 30 06:45:17.696589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.696566 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg"] Oct 30 06:45:17.719539 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.719514 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29"] Oct 30 06:45:17.766468 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.766446 4431 scope.go:122] "RemoveContainer" containerID="072f27dfc794c85e34a50cb0686f1c656f9ee4c9413a515eed95789ab067eefd" Oct 30 06:45:17.791178 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.791096 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c"] Oct 30 06:45:17.812333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.804191 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-645f2b-29889045-lnz9c"] Oct 30 06:45:17.903352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.901140 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl"] Oct 30 06:45:17.905766 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:17.905737 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8d44dd-29889045-h5pkl"] Oct 30 06:45:18.011636 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.011591 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:18.059132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.059107 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:18.116696 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.116665 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-kube-api-access-97hdw\" (UniqueName: \"kubernetes.io/projected/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-kube-api-access-97hdw\") pod \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\" (UID: \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\") " Oct 30 06:45:18.117221 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.116778 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79acdb09-1b01-4201-a885-2d7a90b36371-kube-api-access-78mmc\" (UniqueName: \"kubernetes.io/projected/79acdb09-1b01-4201-a885-2d7a90b36371-kube-api-access-78mmc\") pod \"79acdb09-1b01-4201-a885-2d7a90b36371\" (UID: \"79acdb09-1b01-4201-a885-2d7a90b36371\") " Oct 30 06:45:18.117221 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.116965 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-maas-api-service-ca\") pod \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\" (UID: \"ffff3eb1-6a5f-422c-9a4e-c52892ee3577\") " Oct 30 06:45:18.117221 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.117163 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79acdb09-1b01-4201-a885-2d7a90b36371-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79acdb09-1b01-4201-a885-2d7a90b36371-maas-api-service-ca\") pod \"79acdb09-1b01-4201-a885-2d7a90b36371\" (UID: \"79acdb09-1b01-4201-a885-2d7a90b36371\") " Oct 30 06:45:18.117506 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.117448 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-maas-api-service-ca" pod "ffff3eb1-6a5f-422c-9a4e-c52892ee3577" (UID: "ffff3eb1-6a5f-422c-9a4e-c52892ee3577"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:18.117833 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.117807 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79acdb09-1b01-4201-a885-2d7a90b36371-maas-api-service-ca" pod "79acdb09-1b01-4201-a885-2d7a90b36371" (UID: "79acdb09-1b01-4201-a885-2d7a90b36371"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:18.117989 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.117965 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:18.118067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.117996 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79acdb09-1b01-4201-a885-2d7a90b36371-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:18.119167 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.119140 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79acdb09-1b01-4201-a885-2d7a90b36371-kube-api-access-78mmc" pod "79acdb09-1b01-4201-a885-2d7a90b36371" (UID: "79acdb09-1b01-4201-a885-2d7a90b36371"). InnerVolumeSpecName "kube-api-access-78mmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:18.119262 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.119189 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-kube-api-access-97hdw" pod "ffff3eb1-6a5f-422c-9a4e-c52892ee3577" (UID: "ffff3eb1-6a5f-422c-9a4e-c52892ee3577"). InnerVolumeSpecName "kube-api-access-97hdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:18.219927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.219890 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97hdw\" (UniqueName: \"kubernetes.io/projected/ffff3eb1-6a5f-422c-9a4e-c52892ee3577-kube-api-access-97hdw\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:18.219927 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.219930 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78mmc\" (UniqueName: \"kubernetes.io/projected/79acdb09-1b01-4201-a885-2d7a90b36371-kube-api-access-78mmc\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:18.330286 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.330261 4431 scope.go:122] "RemoveContainer" containerID="a70b024cc8bcd4e5e23d0c32eec32f942d2208ca16440bfb88d3918b7b5fea6b" Oct 30 06:45:18.403141 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.403112 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:18.409842 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.409820 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:18.661419 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.661390 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" event={"ID":"79acdb09-1b01-4201-a885-2d7a90b36371","Type":"ContainerDied","Data":"35db29173b8d83df5de52047933576b3a48cb76f3a3ed6a6d4ee0d2063fd8e72"} Oct 30 06:45:18.661571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.661451 4431 scope.go:122] "RemoveContainer" containerID="9fdee7f790e16b19e184c5a2a83fa633a5632bcfdb8092947fb1d8856c18b297" Oct 30 06:45:18.661571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.661400 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29" Oct 30 06:45:18.664753 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.664724 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" Oct 30 06:45:18.664753 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.664744 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg" event={"ID":"ffff3eb1-6a5f-422c-9a4e-c52892ee3577","Type":"ContainerDied","Data":"3bbba47c346115df83254d6eea15c94e8ac68c2fb0818931626daf5b03ac13ff"} Oct 30 06:45:18.666898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.666877 4431 generic.go:356] "Generic (PLEG): container finished" podID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerID="5eac1710be89c5895ce50c320cbf25c56e8c22c25698e2d717412b9bc87793da" exitCode=60 Oct 30 06:45:18.666970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.666945 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" event={"ID":"1e81e0bf-da55-4013-92ed-04eb35ac1572","Type":"ContainerDied","Data":"5eac1710be89c5895ce50c320cbf25c56e8c22c25698e2d717412b9bc87793da"} Oct 30 06:45:18.667405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.667378 4431 scope.go:122] "RemoveContainer" containerID="5eac1710be89c5895ce50c320cbf25c56e8c22c25698e2d717412b9bc87793da" Oct 30 06:45:18.667717 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:18.667695 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq_odh-ai-gateway-infra(1e81e0bf-da55-4013-92ed-04eb35ac1572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" Oct 30 06:45:18.669204 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.669179 4431 generic.go:356] "Generic (PLEG): container finished" podID="6617a063-2584-41b7-8585-0f5c335fab60" containerID="985e3e417059c905f168c48e55baaa50d251c7d6b2534e03135b05cbec5a76ae" exitCode=60 Oct 30 06:45:18.669295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.669242 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" event={"ID":"6617a063-2584-41b7-8585-0f5c335fab60","Type":"ContainerDied","Data":"985e3e417059c905f168c48e55baaa50d251c7d6b2534e03135b05cbec5a76ae"} Oct 30 06:45:18.669657 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.669639 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:18.669890 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.669866 4431 scope.go:122] "RemoveContainer" containerID="985e3e417059c905f168c48e55baaa50d251c7d6b2534e03135b05cbec5a76ae" Oct 30 06:45:18.670268 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:18.670239 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn_odh-ai-gateway-infra(6617a063-2584-41b7-8585-0f5c335fab60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" podUID="6617a063-2584-41b7-8585-0f5c335fab60" Oct 30 06:45:18.670670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.670653 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:45:18.673983 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.673964 4431 scope.go:122] "RemoveContainer" containerID="a5538f5fd296d4cb8c06665fd5a3c143be808a421530b162252a386b97124336" Oct 30 06:45:18.682943 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.682926 4431 scope.go:122] "RemoveContainer" containerID="04bb4474b85a6f25265d3f4a7498212d4229d3129c280d8a154eab7ec7ed3fa4" Oct 30 06:45:18.696031 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.696007 4431 scope.go:122] "RemoveContainer" containerID="ebe83a9c140e8a7ec9631578f1b5e7926bd5aeea6e4ab77bebfa9c1f59cb6fd5" Oct 30 06:45:18.742214 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.742185 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn"] Oct 30 06:45:18.768362 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.768328 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg"] Oct 30 06:45:18.778000 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.777974 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889045-mhcmg"] Oct 30 06:45:18.849400 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.849285 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29"] Oct 30 06:45:18.853124 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.853098 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6e2548-29889045-f6l29"] Oct 30 06:45:18.917763 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.917735 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv"] Oct 30 06:45:18.968180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:18.968152 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n"] Oct 30 06:45:19.133242 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.133206 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:19.165342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.165289 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:19.235480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.235445 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-kube-api-access-8tc9r\" (UniqueName: \"kubernetes.io/projected/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-kube-api-access-8tc9r\") pod \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\" (UID: \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\") " Oct 30 06:45:19.235671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.235596 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-kube-api-access-t7l9l\" (UniqueName: \"kubernetes.io/projected/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-kube-api-access-t7l9l\") pod \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\" (UID: \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\") " Oct 30 06:45:19.236252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.236228 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-maas-api-service-ca\") pod \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\" (UID: \"30ec2fe4-db3f-456c-86c3-3a9ce49fba37\") " Oct 30 06:45:19.236680 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.236637 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-maas-api-service-ca" pod "30ec2fe4-db3f-456c-86c3-3a9ce49fba37" (UID: "30ec2fe4-db3f-456c-86c3-3a9ce49fba37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:19.236913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.236894 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-maas-api-service-ca\") pod \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\" (UID: \"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6\") " Oct 30 06:45:19.237510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.237391 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-maas-api-service-ca" pod "c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" (UID: "c4eb09e0-cbe8-4865-9718-bd6eb4931bf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:19.238955 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.238287 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:19.239180 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.238971 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:19.240833 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.240731 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-kube-api-access-8tc9r" pod "c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" (UID: "c4eb09e0-cbe8-4865-9718-bd6eb4931bf6"). InnerVolumeSpecName "kube-api-access-8tc9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:19.241088 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.241063 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-kube-api-access-t7l9l" pod "30ec2fe4-db3f-456c-86c3-3a9ce49fba37" (UID: "30ec2fe4-db3f-456c-86c3-3a9ce49fba37"). InnerVolumeSpecName "kube-api-access-t7l9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:19.338929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.338903 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463634d3-21a1-4381-a6c7-45aea8046396" path="/var/lib/kubelet/pods/463634d3-21a1-4381-a6c7-45aea8046396/volumes" Oct 30 06:45:19.339381 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.339367 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" path="/var/lib/kubelet/pods/79acdb09-1b01-4201-a885-2d7a90b36371/volumes" Oct 30 06:45:19.339816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.339802 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" path="/var/lib/kubelet/pods/a8ee227b-e549-4ab1-b046-fdcd9f6b7409/volumes" Oct 30 06:45:19.339952 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.339934 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7l9l\" (UniqueName: \"kubernetes.io/projected/30ec2fe4-db3f-456c-86c3-3a9ce49fba37-kube-api-access-t7l9l\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:19.339988 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.339956 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tc9r\" (UniqueName: \"kubernetes.io/projected/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6-kube-api-access-8tc9r\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:19.340520 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.340505 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" path="/var/lib/kubelet/pods/ffff3eb1-6a5f-422c-9a4e-c52892ee3577/volumes" Oct 30 06:45:19.683014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.682982 4431 generic.go:356] "Generic (PLEG): container finished" podID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerID="06da3a0ee0df2415e83d8478d07e8e53e0cba4aa10a504104c623b3acdd3ebeb" exitCode=60 Oct 30 06:45:19.683172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.683067 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" event={"ID":"409eb369-5875-42cf-a8d1-fec3167d20cd","Type":"ContainerDied","Data":"06da3a0ee0df2415e83d8478d07e8e53e0cba4aa10a504104c623b3acdd3ebeb"} Oct 30 06:45:19.683172 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.683148 4431 scope.go:122] "RemoveContainer" containerID="a70b024cc8bcd4e5e23d0c32eec32f942d2208ca16440bfb88d3918b7b5fea6b" Oct 30 06:45:19.683762 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.683666 4431 scope.go:122] "RemoveContainer" containerID="06da3a0ee0df2415e83d8478d07e8e53e0cba4aa10a504104c623b3acdd3ebeb" Oct 30 06:45:19.684092 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:19.684068 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v_odh-ai-gateway-infra(409eb369-5875-42cf-a8d1-fec3167d20cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" Oct 30 06:45:19.685785 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.685766 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" Oct 30 06:45:19.685872 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.685796 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n" event={"ID":"30ec2fe4-db3f-456c-86c3-3a9ce49fba37","Type":"ContainerDied","Data":"7fa581131c2619e04c65e7c85cf877861d050fa1baac7b75e15583f90fb8b784"} Oct 30 06:45:19.688701 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.688677 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" Oct 30 06:45:19.688776 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.688713 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv" event={"ID":"c4eb09e0-cbe8-4865-9718-bd6eb4931bf6","Type":"ContainerDied","Data":"1ebabaeeba3c3d047cc38cf2b81e3d251b16bc3f0721b434ea2c689a17c61a86"} Oct 30 06:45:19.698536 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.698517 4431 scope.go:122] "RemoveContainer" containerID="d9346eee1007506cfdbe096bc497d9096908244d69b8a393bea920d532bed21e" Oct 30 06:45:19.707777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.707756 4431 scope.go:122] "RemoveContainer" containerID="19e8fff7729d6f4a400cb17651e7b2a6bbca8c26d7572c5bb5aebe26308bd758" Oct 30 06:45:19.831321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.831270 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv"] Oct 30 06:45:19.837140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.837109 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-30fbb6-29889045-vdkzv"] Oct 30 06:45:19.842352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.842329 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq"] Oct 30 06:45:19.869996 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.869966 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n"] Oct 30 06:45:19.874033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.874004 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d8df2-29889045-lmx5n"] Oct 30 06:45:19.891223 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.891198 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:19.956354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.956293 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6617a063-2584-41b7-8585-0f5c335fab60-kube-api-access-4qlcb\" (UniqueName: \"kubernetes.io/projected/6617a063-2584-41b7-8585-0f5c335fab60-kube-api-access-4qlcb\") pod \"6617a063-2584-41b7-8585-0f5c335fab60\" (UID: \"6617a063-2584-41b7-8585-0f5c335fab60\") " Oct 30 06:45:19.966420 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.956474 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6617a063-2584-41b7-8585-0f5c335fab60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617a063-2584-41b7-8585-0f5c335fab60-maas-api-service-ca\") pod \"6617a063-2584-41b7-8585-0f5c335fab60\" (UID: \"6617a063-2584-41b7-8585-0f5c335fab60\") " Oct 30 06:45:19.966420 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.957655 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6617a063-2584-41b7-8585-0f5c335fab60-maas-api-service-ca" pod "6617a063-2584-41b7-8585-0f5c335fab60" (UID: "6617a063-2584-41b7-8585-0f5c335fab60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:19.977253 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:19.977227 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6617a063-2584-41b7-8585-0f5c335fab60-kube-api-access-4qlcb" pod "6617a063-2584-41b7-8585-0f5c335fab60" (UID: "6617a063-2584-41b7-8585-0f5c335fab60"). InnerVolumeSpecName "kube-api-access-4qlcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:20.008817 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.008792 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:20.058078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.058035 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e81e0bf-da55-4013-92ed-04eb35ac1572-kube-api-access-4lrgh\" (UniqueName: \"kubernetes.io/projected/1e81e0bf-da55-4013-92ed-04eb35ac1572-kube-api-access-4lrgh\") pod \"1e81e0bf-da55-4013-92ed-04eb35ac1572\" (UID: \"1e81e0bf-da55-4013-92ed-04eb35ac1572\") " Oct 30 06:45:20.058252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.058158 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e81e0bf-da55-4013-92ed-04eb35ac1572-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e81e0bf-da55-4013-92ed-04eb35ac1572-maas-api-service-ca\") pod \"1e81e0bf-da55-4013-92ed-04eb35ac1572\" (UID: \"1e81e0bf-da55-4013-92ed-04eb35ac1572\") " Oct 30 06:45:20.058918 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.058889 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qlcb\" (UniqueName: \"kubernetes.io/projected/6617a063-2584-41b7-8585-0f5c335fab60-kube-api-access-4qlcb\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.059034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.058924 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6617a063-2584-41b7-8585-0f5c335fab60-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.059034 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.058921 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e81e0bf-da55-4013-92ed-04eb35ac1572-maas-api-service-ca" pod "1e81e0bf-da55-4013-92ed-04eb35ac1572" (UID: "1e81e0bf-da55-4013-92ed-04eb35ac1572"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:20.061001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.060979 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e81e0bf-da55-4013-92ed-04eb35ac1572-kube-api-access-4lrgh" pod "1e81e0bf-da55-4013-92ed-04eb35ac1572" (UID: "1e81e0bf-da55-4013-92ed-04eb35ac1572"). InnerVolumeSpecName "kube-api-access-4lrgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:20.187100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.183085 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrgh\" (UniqueName: \"kubernetes.io/projected/1e81e0bf-da55-4013-92ed-04eb35ac1572-kube-api-access-4lrgh\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.187100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.183122 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e81e0bf-da55-4013-92ed-04eb35ac1572-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.700447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.700417 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" Oct 30 06:45:20.700613 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.700422 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq" event={"ID":"1e81e0bf-da55-4013-92ed-04eb35ac1572","Type":"ContainerDied","Data":"5a6b87463ed6b47cf470e29b91654fbc1999e90a0b13dda619780d28a9b1fd94"} Oct 30 06:45:20.700613 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.700547 4431 scope.go:122] "RemoveContainer" containerID="5eac1710be89c5895ce50c320cbf25c56e8c22c25698e2d717412b9bc87793da" Oct 30 06:45:20.701910 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.701890 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" Oct 30 06:45:20.701985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.701922 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn" event={"ID":"6617a063-2584-41b7-8585-0f5c335fab60","Type":"ContainerDied","Data":"4081f12f79dda9a215607679579b5eeb54a3579ea5045e3ca5cb48e8ae7b01c4"} Oct 30 06:45:20.711498 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.711475 4431 scope.go:122] "RemoveContainer" containerID="985e3e417059c905f168c48e55baaa50d251c7d6b2534e03135b05cbec5a76ae" Oct 30 06:45:20.756695 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.756669 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq"] Oct 30 06:45:20.766292 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.766270 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b8ca49-29889045-ss6bq"] Oct 30 06:45:20.775989 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.775965 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v"] Oct 30 06:45:20.811671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.811635 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn"] Oct 30 06:45:20.815296 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.815270 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889045-mf7fn"] Oct 30 06:45:20.944883 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.944858 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:20.993251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.993188 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409eb369-5875-42cf-a8d1-fec3167d20cd-kube-api-access-sdkjb\" (UniqueName: \"kubernetes.io/projected/409eb369-5875-42cf-a8d1-fec3167d20cd-kube-api-access-sdkjb\") pod \"409eb369-5875-42cf-a8d1-fec3167d20cd\" (UID: \"409eb369-5875-42cf-a8d1-fec3167d20cd\") " Oct 30 06:45:20.993374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.993352 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409eb369-5875-42cf-a8d1-fec3167d20cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409eb369-5875-42cf-a8d1-fec3167d20cd-maas-api-service-ca\") pod \"409eb369-5875-42cf-a8d1-fec3167d20cd\" (UID: \"409eb369-5875-42cf-a8d1-fec3167d20cd\") " Oct 30 06:45:20.993683 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.993655 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409eb369-5875-42cf-a8d1-fec3167d20cd-maas-api-service-ca" pod "409eb369-5875-42cf-a8d1-fec3167d20cd" (UID: "409eb369-5875-42cf-a8d1-fec3167d20cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:45:20.993879 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.993863 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409eb369-5875-42cf-a8d1-fec3167d20cd-maas-api-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:20.995364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:20.995339 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409eb369-5875-42cf-a8d1-fec3167d20cd-kube-api-access-sdkjb" pod "409eb369-5875-42cf-a8d1-fec3167d20cd" (UID: "409eb369-5875-42cf-a8d1-fec3167d20cd"). InnerVolumeSpecName "kube-api-access-sdkjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:45:21.095199 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.095161 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdkjb\" (UniqueName: \"kubernetes.io/projected/409eb369-5875-42cf-a8d1-fec3167d20cd-kube-api-access-sdkjb\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:45:21.342188 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.342079 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" path="/var/lib/kubelet/pods/1e81e0bf-da55-4013-92ed-04eb35ac1572/volumes" Oct 30 06:45:21.343186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.343160 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" path="/var/lib/kubelet/pods/30ec2fe4-db3f-456c-86c3-3a9ce49fba37/volumes" Oct 30 06:45:21.343797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.343775 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6617a063-2584-41b7-8585-0f5c335fab60" path="/var/lib/kubelet/pods/6617a063-2584-41b7-8585-0f5c335fab60/volumes" Oct 30 06:45:21.344787 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.344765 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" path="/var/lib/kubelet/pods/c4eb09e0-cbe8-4865-9718-bd6eb4931bf6/volumes" Oct 30 06:45:21.713192 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.713164 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" Oct 30 06:45:21.713408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.713193 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v" event={"ID":"409eb369-5875-42cf-a8d1-fec3167d20cd","Type":"ContainerDied","Data":"3685cde110e4fe61ee30855fd60ac90cd85a2c0ac0406c10962b27832e84876d"} Oct 30 06:45:21.713408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.713262 4431 scope.go:122] "RemoveContainer" containerID="06da3a0ee0df2415e83d8478d07e8e53e0cba4aa10a504104c623b3acdd3ebeb" Oct 30 06:45:21.771501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.771475 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v"] Oct 30 06:45:21.777483 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:21.777460 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4f3973-29889045-rdf6v"] Oct 30 06:45:22.103438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.103359 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb"] Oct 30 06:45:22.104032 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104018 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerName="cleanup" Oct 30 06:45:22.104089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104033 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerName="cleanup" Oct 30 06:45:22.104089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104047 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerName="cleanup" Oct 30 06:45:22.104089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104052 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerName="cleanup" Oct 30 06:45:22.104089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104062 4431 cpu_manager.go:414] "containerMap: removing container" podUID="463634d3-21a1-4381-a6c7-45aea8046396" containerName="cleanup" Oct 30 06:45:22.104089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104069 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463634d3-21a1-4381-a6c7-45aea8046396" containerName="cleanup" Oct 30 06:45:22.104089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104083 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerName="cleanup" Oct 30 06:45:22.104089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104087 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104094 4431 cpu_manager.go:414] "containerMap: removing container" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104099 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104109 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104114 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104123 4431 cpu_manager.go:414] "containerMap: removing container" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104127 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104138 4431 cpu_manager.go:414] "containerMap: removing container" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104142 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104148 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104152 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104159 4431 cpu_manager.go:414] "containerMap: removing container" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104163 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104168 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6617a063-2584-41b7-8585-0f5c335fab60" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104172 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6617a063-2584-41b7-8585-0f5c335fab60" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104178 4431 cpu_manager.go:414] "containerMap: removing container" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104183 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104191 4431 cpu_manager.go:414] "containerMap: removing container" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104196 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104201 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104205 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104211 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104215 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104221 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6617a063-2584-41b7-8585-0f5c335fab60" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104225 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6617a063-2584-41b7-8585-0f5c335fab60" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104233 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104237 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104249 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6617a063-2584-41b7-8585-0f5c335fab60" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104253 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6617a063-2584-41b7-8585-0f5c335fab60" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104260 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104264 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104272 4431 cpu_manager.go:414] "containerMap: removing container" podUID="463634d3-21a1-4381-a6c7-45aea8046396" containerName="cleanup" Oct 30 06:45:22.104347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104276 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463634d3-21a1-4381-a6c7-45aea8046396" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104380 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104389 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6617a063-2584-41b7-8585-0f5c335fab60" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104395 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104401 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="463634d3-21a1-4381-a6c7-45aea8046396" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104408 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104412 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104418 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104423 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104428 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="463634d3-21a1-4381-a6c7-45aea8046396" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104434 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104440 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104446 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104452 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104458 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104463 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104468 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104472 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104478 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104485 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6617a063-2584-41b7-8585-0f5c335fab60" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104489 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" containerName="cleanup" Oct 30 06:45:22.105240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.104495 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerName="cleanup" Oct 30 06:45:22.109408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.109384 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:22.124651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.124627 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1c148a-dockercfg-4nxwk\"" Oct 30 06:45:22.125225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.125205 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb"] Oct 30 06:45:22.127459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.127437 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1c148a\"" Oct 30 06:45:22.204067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204033 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l"] Oct 30 06:45:22.204831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204813 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerName="cleanup" Oct 30 06:45:22.204831 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204832 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4eb09e0-cbe8-4865-9718-bd6eb4931bf6" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204850 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204856 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204870 4431 cpu_manager.go:414] "containerMap: removing container" podUID="463634d3-21a1-4381-a6c7-45aea8046396" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204876 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463634d3-21a1-4381-a6c7-45aea8046396" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204892 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204897 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ee227b-e549-4ab1-b046-fdcd9f6b7409" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204905 4431 cpu_manager.go:414] "containerMap: removing container" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204909 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204931 4431 cpu_manager.go:414] "containerMap: removing container" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerName="cleanup" Oct 30 06:45:22.204937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204935 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" containerName="cleanup" Oct 30 06:45:22.205227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204944 4431 cpu_manager.go:414] "containerMap: removing container" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerName="cleanup" Oct 30 06:45:22.205227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.204949 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerName="cleanup" Oct 30 06:45:22.205227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.205037 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffff3eb1-6a5f-422c-9a4e-c52892ee3577" containerName="cleanup" Oct 30 06:45:22.205227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.205048 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="79acdb09-1b01-4201-a885-2d7a90b36371" containerName="cleanup" Oct 30 06:45:22.205227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.205055 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6617a063-2584-41b7-8585-0f5c335fab60" containerName="cleanup" Oct 30 06:45:22.205227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.205063 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e81e0bf-da55-4013-92ed-04eb35ac1572" containerName="cleanup" Oct 30 06:45:22.205227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.205068 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ec2fe4-db3f-456c-86c3-3a9ce49fba37" containerName="cleanup" Oct 30 06:45:22.205227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.205076 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="463634d3-21a1-4381-a6c7-45aea8046396" containerName="cleanup" Oct 30 06:45:22.208733 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.208713 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:45:22.211057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.211032 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3879a-73ba-46f3-b878-5cd4d76c286e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb\" (UID: \"22c3879a-73ba-46f3-b878-5cd4d76c286e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:22.211184 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.211169 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28h67\" (UniqueName: \"kubernetes.io/projected/22c3879a-73ba-46f3-b878-5cd4d76c286e-kube-api-access-28h67\") pod \"payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb\" (UID: \"22c3879a-73ba-46f3-b878-5cd4d76c286e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:22.225959 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.225931 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l"] Oct 30 06:45:22.342325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.315971 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3879a-73ba-46f3-b878-5cd4d76c286e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb\" (UID: \"22c3879a-73ba-46f3-b878-5cd4d76c286e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:22.342325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.316045 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l\" (UID: \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:45:22.342325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.316111 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqqbt\" (UniqueName: \"kubernetes.io/projected/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-kube-api-access-sqqbt\") pod \"payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l\" (UID: \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:45:22.342325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.316171 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28h67\" (UniqueName: \"kubernetes.io/projected/22c3879a-73ba-46f3-b878-5cd4d76c286e-kube-api-access-28h67\") pod \"payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb\" (UID: \"22c3879a-73ba-46f3-b878-5cd4d76c286e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:22.387390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.345436 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3879a-73ba-46f3-b878-5cd4d76c286e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb\" (UID: \"22c3879a-73ba-46f3-b878-5cd4d76c286e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:22.424856 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.424811 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l\" (UID: \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:45:22.446472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.428416 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqqbt\" (UniqueName: \"kubernetes.io/projected/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-kube-api-access-sqqbt\") pod \"payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l\" (UID: \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:45:22.446472 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.426788 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l\" (UID: \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:45:22.498469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.498435 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28h67\" (UniqueName: \"kubernetes.io/projected/22c3879a-73ba-46f3-b878-5cd4d76c286e-kube-api-access-28h67\") pod \"payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb\" (UID: \"22c3879a-73ba-46f3-b878-5cd4d76c286e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:22.609221 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.596097 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqqbt\" (UniqueName: \"kubernetes.io/projected/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-kube-api-access-sqqbt\") pod \"payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l\" (UID: \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:45:22.727901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.723995 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:22.825259 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:22.825225 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:45:23.088848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.088820 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb"] Oct 30 06:45:23.090000 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:23.089968 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c3879a_73ba_46f3_b878_5cd4d76c286e.slice/crio-c0616f342a419c8fa8d9f5c74ba4fd4e0b012d327b6110555318d1c9ab34cc70 WatchSource:0}: Error finding container c0616f342a419c8fa8d9f5c74ba4fd4e0b012d327b6110555318d1c9ab34cc70: Status 404 returned error can't find the container with id c0616f342a419c8fa8d9f5c74ba4fd4e0b012d327b6110555318d1c9ab34cc70 Oct 30 06:45:23.179026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.179004 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l"] Oct 30 06:45:23.180680 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:45:23.180640 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a60cbf4_4f6e_4bef_810a_6d72d11feaa3.slice/crio-28e20f819226e12bca103b3ec08e97dafc4c05f8a584355ec4e4f71ad3b3165d WatchSource:0}: Error finding container 28e20f819226e12bca103b3ec08e97dafc4c05f8a584355ec4e4f71ad3b3165d: Status 404 returned error can't find the container with id 28e20f819226e12bca103b3ec08e97dafc4c05f8a584355ec4e4f71ad3b3165d Oct 30 06:45:23.340211 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.340143 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409eb369-5875-42cf-a8d1-fec3167d20cd" path="/var/lib/kubelet/pods/409eb369-5875-42cf-a8d1-fec3167d20cd/volumes" Oct 30 06:45:23.738244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.738206 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" event={"ID":"22c3879a-73ba-46f3-b878-5cd4d76c286e","Type":"ContainerStarted","Data":"44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9"} Oct 30 06:45:23.738704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.738260 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" event={"ID":"22c3879a-73ba-46f3-b878-5cd4d76c286e","Type":"ContainerStarted","Data":"c0616f342a419c8fa8d9f5c74ba4fd4e0b012d327b6110555318d1c9ab34cc70"} Oct 30 06:45:23.738704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.738328 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:23.740045 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.740022 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" event={"ID":"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3","Type":"ContainerStarted","Data":"77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab"} Oct 30 06:45:23.740175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.740063 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" event={"ID":"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3","Type":"ContainerStarted","Data":"28e20f819226e12bca103b3ec08e97dafc4c05f8a584355ec4e4f71ad3b3165d"} Oct 30 06:45:23.740175 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.740092 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:45:23.780983 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.780929 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" podStartSLOduration=1.78091157 podStartE2EDuration="1.78091157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:23.779250712 +0000 UTC m=+3132.585772461" watchObservedRunningTime="2026-10-30 06:45:23.78091157 +0000 UTC m=+3132.587433319" Oct 30 06:45:23.820333 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:23.820250 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" podStartSLOduration=1.820236531 podStartE2EDuration="1.820236531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:45:23.818582474 +0000 UTC m=+3132.625104223" watchObservedRunningTime="2026-10-30 06:45:23.820236531 +0000 UTC m=+3132.626758286" Oct 30 06:45:33.701527 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:45:33.701496 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:45:34.752687 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:34.752655 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:45:34.753119 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:45:34.752891 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:46:03.082819 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.082782 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng"] Oct 30 06:46:03.087068 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.087050 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.106749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.106707 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng"] Oct 30 06:46:03.115345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.113409 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1c148a-serving-cert\"" Oct 30 06:46:03.203238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.203209 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvdx\" (UniqueName: \"kubernetes.io/projected/3c067b93-6503-40a6-9d51-bc566d49d33d-kube-api-access-2lvdx\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.203464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.203284 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.203464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.203362 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c067b93-6503-40a6-9d51-bc566d49d33d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.203464 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.203450 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-metrics-certs\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.305417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.305385 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvdx\" (UniqueName: \"kubernetes.io/projected/3c067b93-6503-40a6-9d51-bc566d49d33d-kube-api-access-2lvdx\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.305577 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.305461 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.305577 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.305477 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c067b93-6503-40a6-9d51-bc566d49d33d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.305577 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.305505 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-metrics-certs\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.306179 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.306148 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c067b93-6503-40a6-9d51-bc566d49d33d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.308762 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.308612 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-metrics-certs\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.308762 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.308668 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.349369 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.340110 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvdx\" (UniqueName: \"kubernetes.io/projected/3c067b93-6503-40a6-9d51-bc566d49d33d-kube-api-access-2lvdx\") pod \"maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.404618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.404579 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:03.702689 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:46:03.702655 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:46:03.741892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.741871 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng"] Oct 30 06:46:03.743490 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:46:03.743455 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c067b93_6503_40a6_9d51_bc566d49d33d.slice/crio-0c7d4d198f1108584417c20dd2ec49aff7e11e3af9c486b47f967e5e9aa4d280 WatchSource:0}: Error finding container 0c7d4d198f1108584417c20dd2ec49aff7e11e3af9c486b47f967e5e9aa4d280: Status 404 returned error can't find the container with id 0c7d4d198f1108584417c20dd2ec49aff7e11e3af9c486b47f967e5e9aa4d280 Oct 30 06:46:03.744901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:03.744883 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:46:04.073647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:04.073563 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" event={"ID":"3c067b93-6503-40a6-9d51-bc566d49d33d","Type":"ContainerStarted","Data":"0c7d4d198f1108584417c20dd2ec49aff7e11e3af9c486b47f967e5e9aa4d280"} Oct 30 06:46:05.083168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:05.083134 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" event={"ID":"3c067b93-6503-40a6-9d51-bc566d49d33d","Type":"ContainerStarted","Data":"86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558"} Oct 30 06:46:05.083811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:05.083243 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:05.125875 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:05.125835 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" podStartSLOduration=1.716901933 podStartE2EDuration="2.125822868s" totalImagesPullingTime="408.920935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:46:05.125270874 +0000 UTC m=+3173.931792623" watchObservedRunningTime="2026-10-30 06:46:05.125822868 +0000 UTC m=+3173.932344615" Oct 30 06:46:11.096691 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:11.096662 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:46:33.701190 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:46:33.701153 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:46:34.710492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:34.710449 4431 patch_prober.go:29] interesting pod/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:46:34.710913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:34.710532 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:46:37.445980 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:37.445946 4431 generic.go:356] "Generic (PLEG): container finished" podID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerID="8da47eee2ac4486c631fef1c07b0dd1c0aac445035677a776160a8db06417b29" exitCode=0 Oct 30 06:46:37.446562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:37.446164 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" event={"ID":"9773d54c-d0fa-441b-b9b7-fe731d630d3a","Type":"ContainerDied","Data":"8da47eee2ac4486c631fef1c07b0dd1c0aac445035677a776160a8db06417b29"} Oct 30 06:46:37.446845 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:37.446825 4431 scope.go:122] "RemoveContainer" containerID="8da47eee2ac4486c631fef1c07b0dd1c0aac445035677a776160a8db06417b29" Oct 30 06:46:37.448838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:37.448806 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:46:37.448956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:37.448883 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:46:37.448956 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:37.448917 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:46:38.454844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:38.454814 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" event={"ID":"9773d54c-d0fa-441b-b9b7-fe731d630d3a","Type":"ContainerStarted","Data":"737860eeb84ff3f1d6deb4f6f02711278d9993305023543f4f3578d79c29ed85"} Oct 30 06:46:39.403156 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:39.403118 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:46:39.403379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:39.403352 4431 patch_prober.go:29] interesting pod/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 06:46:39.403438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:39.403417 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 06:46:40.403061 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:40.403028 4431 patch_prober.go:29] interesting pod/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 06:46:40.403508 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:40.403100 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 06:46:41.403561 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:41.403531 4431 patch_prober.go:29] interesting pod/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 06:46:41.403972 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:41.403588 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 06:46:42.403133 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:42.403098 4431 patch_prober.go:29] interesting pod/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 06:46:42.403377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:42.403154 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 06:46:43.403391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:43.403353 4431 patch_prober.go:29] interesting pod/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 06:46:43.403913 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:43.403418 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 06:46:44.456434 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:44.456401 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:46:44.457784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:44.457754 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:46:44.458606 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:46:44.458581 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:47:03.701224 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:47:03.701184 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:47:13.612573 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.612540 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-d5q7w"] Oct 30 06:47:13.617295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.617273 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.636058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.636017 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-d5q7w"] Oct 30 06:47:13.642229 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.642205 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 06:47:13.642419 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.642205 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 06:47:13.642555 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.642234 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c537891062d6\"" Oct 30 06:47:13.643982 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.643962 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 06:47:13.644145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.644124 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-8z4m5\"" Oct 30 06:47:13.644241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.644211 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8b3017c11d13\"" Oct 30 06:47:13.644355 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.644294 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 06:47:13.706190 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.706160 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-routing-overlay-volume\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.706391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.706280 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-provider-credentials\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.706391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.706380 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-plugins-config-volume\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.706480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.706413 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2sn\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-kube-api-access-mb2sn\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.808252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.808217 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-provider-credentials\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.808435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.808273 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-plugins-config-volume\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.808435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.808314 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2sn\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-kube-api-access-mb2sn\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.808557 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.808541 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-routing-overlay-volume\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.809148 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.809123 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-routing-overlay-volume\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.809277 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.809147 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-plugins-config-volume\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.810808 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.810786 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-provider-credentials\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.831218 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.831197 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2sn\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-kube-api-access-mb2sn\") pod \"payload-processing-external-model-578778c84f-d5q7w\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:13.932609 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:13.932516 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:14.478573 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:14.473222 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-d5q7w"] Oct 30 06:47:15.117609 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:15.117570 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" event={"ID":"09a12623-f17f-4bdd-9198-ea086b422922","Type":"ContainerStarted","Data":"8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747"} Oct 30 06:47:15.118071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:15.117619 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" event={"ID":"09a12623-f17f-4bdd-9198-ea086b422922","Type":"ContainerStarted","Data":"8dd315fd78b8b1645929628b0d4f0228ff2c624a69f1795f8ac4fcbce2c02723"} Oct 30 06:47:15.118071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:15.117647 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:15.162843 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:15.162779 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" podStartSLOduration=2.162764253 podStartE2EDuration="2.162764253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:15.158825532 +0000 UTC m=+3243.965347281" watchObservedRunningTime="2026-10-30 06:47:15.162764253 +0000 UTC m=+3243.969286001" Oct 30 06:47:26.130022 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:26.129991 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:47:26.548681 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:26.548652 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd"] Oct 30 06:47:26.548881 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:26.548859 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerName="istio-proxy" containerID="cri-o://6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae" gracePeriod=30 Oct 30 06:47:27.207080 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:27.207048 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8"] Oct 30 06:47:27.207542 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:27.207234 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerName="istio-proxy" containerID="cri-o://cfac97b48f5fbb0a07ea5128676837c1a261d09b6e7fc834f8d288e03274cd64" gracePeriod=30 Oct 30 06:47:27.934237 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:27.934199 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx"] Oct 30 06:47:27.934462 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:27.934430 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" containerID="cri-o://f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07" gracePeriod=30 Oct 30 06:47:31.830063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.829996 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:47:31.965089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965058 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-kube-api-access-hzx6f\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-kube-api-access-hzx6f\") pod \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " Oct 30 06:47:31.965283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965110 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-socket\") pod \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " Oct 30 06:47:31.965283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965160 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-podinfo\") pod \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " Oct 30 06:47:31.965283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965207 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-envoy\") pod \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " Oct 30 06:47:31.965283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965268 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-certs\") pod \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " Oct 30 06:47:31.965517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965399 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-data\") pod \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " Oct 30 06:47:31.965517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965446 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-token\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-token\") pod \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " Oct 30 06:47:31.965517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965489 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-credential-socket\") pod \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " Oct 30 06:47:31.965517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965475 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-socket" pod "6a2e0884-320a-4cf8-ba2a-55d22ff54988" (UID: "6a2e0884-320a-4cf8-ba2a-55d22ff54988"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.965738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965536 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-certs" pod "6a2e0884-320a-4cf8-ba2a-55d22ff54988" (UID: "6a2e0884-320a-4cf8-ba2a-55d22ff54988"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.965738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965584 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istiod-ca-cert\") pod \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\" (UID: \"6a2e0884-320a-4cf8-ba2a-55d22ff54988\") " Oct 30 06:47:31.965738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965694 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-data" pod "6a2e0884-320a-4cf8-ba2a-55d22ff54988" (UID: "6a2e0884-320a-4cf8-ba2a-55d22ff54988"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.965898 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.965880 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-credential-socket" pod "6a2e0884-320a-4cf8-ba2a-55d22ff54988" (UID: "6a2e0884-320a-4cf8-ba2a-55d22ff54988"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.966100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.966073 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istiod-ca-cert" pod "6a2e0884-320a-4cf8-ba2a-55d22ff54988" (UID: "6a2e0884-320a-4cf8-ba2a-55d22ff54988"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:31.966295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.966278 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.966374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.966317 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.966374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.966332 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.966374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.966346 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.966374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.966362 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:31.967618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.967594 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-podinfo" pod "6a2e0884-320a-4cf8-ba2a-55d22ff54988" (UID: "6a2e0884-320a-4cf8-ba2a-55d22ff54988"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:31.967692 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.967635 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-envoy" pod "6a2e0884-320a-4cf8-ba2a-55d22ff54988" (UID: "6a2e0884-320a-4cf8-ba2a-55d22ff54988"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:31.967909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.967878 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-kube-api-access-hzx6f" pod "6a2e0884-320a-4cf8-ba2a-55d22ff54988" (UID: "6a2e0884-320a-4cf8-ba2a-55d22ff54988"). InnerVolumeSpecName "kube-api-access-hzx6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:31.967909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:31.967890 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-token" pod "6a2e0884-320a-4cf8-ba2a-55d22ff54988" (UID: "6a2e0884-320a-4cf8-ba2a-55d22ff54988"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.067892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.067862 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.067892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.067888 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzx6f\" (UniqueName: \"kubernetes.io/projected/6a2e0884-320a-4cf8-ba2a-55d22ff54988-kube-api-access-hzx6f\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.067892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.067901 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.068131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.067914 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2e0884-320a-4cf8-ba2a-55d22ff54988-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.127338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.127228 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c"] Oct 30 06:47:32.127513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.127436 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" containerID="cri-o://192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e" gracePeriod=30 Oct 30 06:47:32.338932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.338903 4431 generic.go:356] "Generic (PLEG): container finished" podID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerID="6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae" exitCode=0 Oct 30 06:47:32.339077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.338957 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" Oct 30 06:47:32.339077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.338990 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" event={"ID":"6a2e0884-320a-4cf8-ba2a-55d22ff54988","Type":"ContainerDied","Data":"6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae"} Oct 30 06:47:32.339077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.339055 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd" event={"ID":"6a2e0884-320a-4cf8-ba2a-55d22ff54988","Type":"ContainerDied","Data":"01661b86dfa6fd01b5918c8bde16e293e52897df14fed8798350492e22f9f1dd"} Oct 30 06:47:32.339257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.339090 4431 scope.go:122] "RemoveContainer" containerID="6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae" Oct 30 06:47:32.341050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.341019 4431 generic.go:356] "Generic (PLEG): container finished" podID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerID="cfac97b48f5fbb0a07ea5128676837c1a261d09b6e7fc834f8d288e03274cd64" exitCode=0 Oct 30 06:47:32.341158 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.341066 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" event={"ID":"12ee2fcd-1dfb-47d6-b7aa-395160feb05a","Type":"ContainerDied","Data":"cfac97b48f5fbb0a07ea5128676837c1a261d09b6e7fc834f8d288e03274cd64"} Oct 30 06:47:32.350238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.350103 4431 scope.go:122] "RemoveContainer" containerID="6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae" Oct 30 06:47:32.350554 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:47:32.350529 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae\": container with ID starting with 6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae not found: ID does not exist" containerID="6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae" Oct 30 06:47:32.350677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.350561 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae"} err="failed to get container status \"6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae\": rpc error: code = NotFound desc = could not find container \"6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae\": container with ID starting with 6381e2f1cef3a93a8ccbc736894056cf6677c633aa4e7b9ffa081fb70d319aae not found: ID does not exist" Oct 30 06:47:32.396828 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.396772 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd"] Oct 30 06:47:32.403076 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.403043 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d8df2-openshift-default-5b64664cd5-tlqmd"] Oct 30 06:47:32.469624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.469593 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:47:32.573348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573258 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-kube-api-access-dj4dh\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-kube-api-access-dj4dh\") pod \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " Oct 30 06:47:32.573348 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573320 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-credential-socket\") pod \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " Oct 30 06:47:32.573523 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573372 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-socket\") pod \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " Oct 30 06:47:32.573523 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573426 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-podinfo\") pod \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " Oct 30 06:47:32.573784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573543 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-envoy\") pod \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " Oct 30 06:47:32.573784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573645 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-data\") pod \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " Oct 30 06:47:32.573784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573680 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-socket" pod "12ee2fcd-1dfb-47d6-b7aa-395160feb05a" (UID: "12ee2fcd-1dfb-47d6-b7aa-395160feb05a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.573784 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573700 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-credential-socket" pod "12ee2fcd-1dfb-47d6-b7aa-395160feb05a" (UID: "12ee2fcd-1dfb-47d6-b7aa-395160feb05a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.574087 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573786 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-token\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-token\") pod \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " Oct 30 06:47:32.574087 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573825 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-certs\") pod \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " Oct 30 06:47:32.574087 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573898 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istiod-ca-cert\") pod \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\" (UID: \"12ee2fcd-1dfb-47d6-b7aa-395160feb05a\") " Oct 30 06:47:32.574087 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.573902 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-data" pod "12ee2fcd-1dfb-47d6-b7aa-395160feb05a" (UID: "12ee2fcd-1dfb-47d6-b7aa-395160feb05a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.574087 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.574080 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-certs" pod "12ee2fcd-1dfb-47d6-b7aa-395160feb05a" (UID: "12ee2fcd-1dfb-47d6-b7aa-395160feb05a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.574694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.574653 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.574694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.574684 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.575519 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.574700 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.575519 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.574714 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.575918 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.575895 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istiod-ca-cert" pod "12ee2fcd-1dfb-47d6-b7aa-395160feb05a" (UID: "12ee2fcd-1dfb-47d6-b7aa-395160feb05a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:32.576967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.576906 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-podinfo" pod "12ee2fcd-1dfb-47d6-b7aa-395160feb05a" (UID: "12ee2fcd-1dfb-47d6-b7aa-395160feb05a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:32.576967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.576917 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-token" pod "12ee2fcd-1dfb-47d6-b7aa-395160feb05a" (UID: "12ee2fcd-1dfb-47d6-b7aa-395160feb05a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.577131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.576994 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-kube-api-access-dj4dh" pod "12ee2fcd-1dfb-47d6-b7aa-395160feb05a" (UID: "12ee2fcd-1dfb-47d6-b7aa-395160feb05a"). InnerVolumeSpecName "kube-api-access-dj4dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.577131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.577049 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-envoy" pod "12ee2fcd-1dfb-47d6-b7aa-395160feb05a" (UID: "12ee2fcd-1dfb-47d6-b7aa-395160feb05a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.675815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.675742 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.675815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.675768 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.675815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.675777 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.675815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.675786 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj4dh\" (UniqueName: \"kubernetes.io/projected/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-kube-api-access-dj4dh\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.675815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:32.675795 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ee2fcd-1dfb-47d6-b7aa-395160feb05a-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.267389 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.267353 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:47:33.341838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.341796 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" path="/var/lib/kubelet/pods/6a2e0884-320a-4cf8-ba2a-55d22ff54988/volumes" Oct 30 06:47:33.354110 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.354074 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" event={"ID":"12ee2fcd-1dfb-47d6-b7aa-395160feb05a","Type":"ContainerDied","Data":"a5cba46ce259ad43e37eaac99b7449275e709baa0813701278f60a7e5155525a"} Oct 30 06:47:33.354238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.354147 4431 scope.go:122] "RemoveContainer" containerID="cfac97b48f5fbb0a07ea5128676837c1a261d09b6e7fc834f8d288e03274cd64" Oct 30 06:47:33.354337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.354292 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8" Oct 30 06:47:33.356579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.356556 4431 generic.go:356] "Generic (PLEG): container finished" podID="8e912eea-45ff-47e0-b15b-165a912ce731" containerID="f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07" exitCode=0 Oct 30 06:47:33.356702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.356656 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" event={"ID":"8e912eea-45ff-47e0-b15b-165a912ce731","Type":"ContainerDied","Data":"f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07"} Oct 30 06:47:33.356702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.356697 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" event={"ID":"8e912eea-45ff-47e0-b15b-165a912ce731","Type":"ContainerDied","Data":"7c1cb126b525be1783e3f7be246538c9ad04e31f2c1cbd835abb1925b12f7533"} Oct 30 06:47:33.356821 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.356774 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx" Oct 30 06:47:33.374786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.374764 4431 scope.go:122] "RemoveContainer" containerID="f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07" Oct 30 06:47:33.386920 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.386902 4431 scope.go:122] "RemoveContainer" containerID="f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07" Oct 30 06:47:33.387198 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:47:33.387178 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07\": container with ID starting with f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07 not found: ID does not exist" containerID="f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07" Oct 30 06:47:33.387283 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.387204 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07"} err="failed to get container status \"f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07\": rpc error: code = NotFound desc = could not find container \"f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07\": container with ID starting with f7667684b1e36c7933f27ede23edb0a93fddb7d1275355fec10ffc7d645a7f07 not found: ID does not exist" Oct 30 06:47:33.401099 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401077 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-certs\") pod \"8e912eea-45ff-47e0-b15b-165a912ce731\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " Oct 30 06:47:33.401202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401152 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-credential-socket\") pod \"8e912eea-45ff-47e0-b15b-165a912ce731\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " Oct 30 06:47:33.401247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401211 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e912eea-45ff-47e0-b15b-165a912ce731-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e912eea-45ff-47e0-b15b-165a912ce731-istio-podinfo\") pod \"8e912eea-45ff-47e0-b15b-165a912ce731\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " Oct 30 06:47:33.401332 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401293 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-envoy\") pod \"8e912eea-45ff-47e0-b15b-165a912ce731\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " Oct 30 06:47:33.401412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401349 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-istio-token\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-istio-token\") pod \"8e912eea-45ff-47e0-b15b-165a912ce731\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " Oct 30 06:47:33.401412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401380 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-kube-api-access-p6szt\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-kube-api-access-p6szt\") pod \"8e912eea-45ff-47e0-b15b-165a912ce731\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " Oct 30 06:47:33.401412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401361 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-certs" pod "8e912eea-45ff-47e0-b15b-165a912ce731" (UID: "8e912eea-45ff-47e0-b15b-165a912ce731"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.401585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401422 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-credential-socket" pod "8e912eea-45ff-47e0-b15b-165a912ce731" (UID: "8e912eea-45ff-47e0-b15b-165a912ce731"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.401585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401462 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-data\") pod \"8e912eea-45ff-47e0-b15b-165a912ce731\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " Oct 30 06:47:33.401585 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401567 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-socket\") pod \"8e912eea-45ff-47e0-b15b-165a912ce731\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " Oct 30 06:47:33.401797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401619 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e912eea-45ff-47e0-b15b-165a912ce731-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e912eea-45ff-47e0-b15b-165a912ce731-istiod-ca-cert\") pod \"8e912eea-45ff-47e0-b15b-165a912ce731\" (UID: \"8e912eea-45ff-47e0-b15b-165a912ce731\") " Oct 30 06:47:33.401797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401709 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-data" pod "8e912eea-45ff-47e0-b15b-165a912ce731" (UID: "8e912eea-45ff-47e0-b15b-165a912ce731"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.401936 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.401840 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-socket" pod "8e912eea-45ff-47e0-b15b-165a912ce731" (UID: "8e912eea-45ff-47e0-b15b-165a912ce731"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.402126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.402098 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e912eea-45ff-47e0-b15b-165a912ce731-istiod-ca-cert" pod "8e912eea-45ff-47e0-b15b-165a912ce731" (UID: "8e912eea-45ff-47e0-b15b-165a912ce731"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:33.402272 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.402255 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.402364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.402278 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.402364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.402292 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e912eea-45ff-47e0-b15b-165a912ce731-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.402364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.402322 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.402364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.402336 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.403672 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.403650 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-envoy" pod "8e912eea-45ff-47e0-b15b-165a912ce731" (UID: "8e912eea-45ff-47e0-b15b-165a912ce731"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.403762 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.403668 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-istio-token" pod "8e912eea-45ff-47e0-b15b-165a912ce731" (UID: "8e912eea-45ff-47e0-b15b-165a912ce731"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:33.403938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.403918 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e912eea-45ff-47e0-b15b-165a912ce731-istio-podinfo" pod "8e912eea-45ff-47e0-b15b-165a912ce731" (UID: "8e912eea-45ff-47e0-b15b-165a912ce731"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:33.404012 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.403948 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-kube-api-access-p6szt" pod "8e912eea-45ff-47e0-b15b-165a912ce731" (UID: "8e912eea-45ff-47e0-b15b-165a912ce731"). InnerVolumeSpecName "kube-api-access-p6szt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:33.422685 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.422663 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8"] Oct 30 06:47:33.429693 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.429669 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-5dtg8"] Oct 30 06:47:33.504568 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.504490 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e912eea-45ff-47e0-b15b-165a912ce731-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.504568 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.504519 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e912eea-45ff-47e0-b15b-165a912ce731-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.504568 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.504533 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.504568 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.504547 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6szt\" (UniqueName: \"kubernetes.io/projected/8e912eea-45ff-47e0-b15b-165a912ce731-kube-api-access-p6szt\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.702295 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:47:33.702260 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:47:33.703946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.703921 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx"] Oct 30 06:47:33.715292 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:33.715266 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-xcbkx"] Oct 30 06:47:34.248345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:34.248284 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:47:34.248507 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:34.248366 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:47:35.443885 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:35.443844 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" path="/var/lib/kubelet/pods/12ee2fcd-1dfb-47d6-b7aa-395160feb05a/volumes" Oct 30 06:47:35.456045 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:35.454773 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" path="/var/lib/kubelet/pods/8e912eea-45ff-47e0-b15b-165a912ce731/volumes" Oct 30 06:47:37.397950 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.397894 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:47:37.456709 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.456673 4431 generic.go:356] "Generic (PLEG): container finished" podID="aebc7419-b24e-42f5-9107-fe3143d88963" containerID="192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e" exitCode=0 Oct 30 06:47:37.456914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.456736 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" Oct 30 06:47:37.456914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.456753 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" event={"ID":"aebc7419-b24e-42f5-9107-fe3143d88963","Type":"ContainerDied","Data":"192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e"} Oct 30 06:47:37.456914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.456809 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c" event={"ID":"aebc7419-b24e-42f5-9107-fe3143d88963","Type":"ContainerDied","Data":"98876fdb1f1eb71b8a28841f7b424ec81eb114f587e801b0d0131d070591f99b"} Oct 30 06:47:37.456914 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.456847 4431 scope.go:122] "RemoveContainer" containerID="192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e" Oct 30 06:47:37.469486 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.469461 4431 scope.go:122] "RemoveContainer" containerID="192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e" Oct 30 06:47:37.469868 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:47:37.469825 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e\": container with ID starting with 192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e not found: ID does not exist" containerID="192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e" Oct 30 06:47:37.470038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.469864 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e"} err="failed to get container status \"192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e\": rpc error: code = NotFound desc = could not find container \"192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e\": container with ID starting with 192d75f258474e4affe08841142dab68a77f75fcca8b8eef25fb7915ee1b9d5e not found: ID does not exist" Oct 30 06:47:37.509518 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.509484 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-envoy\") pod \"aebc7419-b24e-42f5-9107-fe3143d88963\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " Oct 30 06:47:37.509708 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.509551 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-credential-socket\") pod \"aebc7419-b24e-42f5-9107-fe3143d88963\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " Oct 30 06:47:37.509708 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.509584 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-istio-token\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-istio-token\") pod \"aebc7419-b24e-42f5-9107-fe3143d88963\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " Oct 30 06:47:37.509708 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.509653 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-socket\") pod \"aebc7419-b24e-42f5-9107-fe3143d88963\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " Oct 30 06:47:37.509916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.509724 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-data\") pod \"aebc7419-b24e-42f5-9107-fe3143d88963\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " Oct 30 06:47:37.509916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.509809 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aebc7419-b24e-42f5-9107-fe3143d88963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebc7419-b24e-42f5-9107-fe3143d88963-istio-podinfo\") pod \"aebc7419-b24e-42f5-9107-fe3143d88963\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " Oct 30 06:47:37.509916 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.509892 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-certs\") pod \"aebc7419-b24e-42f5-9107-fe3143d88963\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " Oct 30 06:47:37.510077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.509920 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aebc7419-b24e-42f5-9107-fe3143d88963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebc7419-b24e-42f5-9107-fe3143d88963-istiod-ca-cert\") pod \"aebc7419-b24e-42f5-9107-fe3143d88963\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " Oct 30 06:47:37.510077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.509985 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-kube-api-access-6vz9g\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-kube-api-access-6vz9g\") pod \"aebc7419-b24e-42f5-9107-fe3143d88963\" (UID: \"aebc7419-b24e-42f5-9107-fe3143d88963\") " Oct 30 06:47:37.510077 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.510011 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-socket" pod "aebc7419-b24e-42f5-9107-fe3143d88963" (UID: "aebc7419-b24e-42f5-9107-fe3143d88963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:37.511414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.510513 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-certs" pod "aebc7419-b24e-42f5-9107-fe3143d88963" (UID: "aebc7419-b24e-42f5-9107-fe3143d88963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:37.511414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.510668 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-credential-socket" pod "aebc7419-b24e-42f5-9107-fe3143d88963" (UID: "aebc7419-b24e-42f5-9107-fe3143d88963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:37.511414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.510796 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-data" pod "aebc7419-b24e-42f5-9107-fe3143d88963" (UID: "aebc7419-b24e-42f5-9107-fe3143d88963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:37.511414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.510875 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aebc7419-b24e-42f5-9107-fe3143d88963-istiod-ca-cert" pod "aebc7419-b24e-42f5-9107-fe3143d88963" (UID: "aebc7419-b24e-42f5-9107-fe3143d88963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:37.511414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.511022 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:37.511414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.511043 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aebc7419-b24e-42f5-9107-fe3143d88963-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:37.511414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.511057 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:37.511414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.511073 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:37.511414 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.511088 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:37.512386 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.512356 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-istio-token" pod "aebc7419-b24e-42f5-9107-fe3143d88963" (UID: "aebc7419-b24e-42f5-9107-fe3143d88963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:37.512524 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.512502 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-envoy" pod "aebc7419-b24e-42f5-9107-fe3143d88963" (UID: "aebc7419-b24e-42f5-9107-fe3143d88963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:37.512649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.512620 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-kube-api-access-6vz9g" pod "aebc7419-b24e-42f5-9107-fe3143d88963" (UID: "aebc7419-b24e-42f5-9107-fe3143d88963"). InnerVolumeSpecName "kube-api-access-6vz9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:37.514394 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.514360 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aebc7419-b24e-42f5-9107-fe3143d88963-istio-podinfo" pod "aebc7419-b24e-42f5-9107-fe3143d88963" (UID: "aebc7419-b24e-42f5-9107-fe3143d88963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:37.612264 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.612230 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aebc7419-b24e-42f5-9107-fe3143d88963-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:37.612264 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.612258 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:37.612264 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.612267 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aebc7419-b24e-42f5-9107-fe3143d88963-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:37.612516 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.612276 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vz9g\" (UniqueName: \"kubernetes.io/projected/aebc7419-b24e-42f5-9107-fe3143d88963-kube-api-access-6vz9g\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:47:37.813864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.813833 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c"] Oct 30 06:47:37.817177 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:37.817152 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pml6c"] Oct 30 06:47:39.339092 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:39.339060 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" path="/var/lib/kubelet/pods/aebc7419-b24e-42f5-9107-fe3143d88963/volumes" Oct 30 06:47:43.844941 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:43.844895 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:47:43.879498 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:43.879455 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" podUID="4199dace-66ce-4982-9ff6-629345b077ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:47:43.948331 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:43.947527 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 30 06:47:43.977319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:43.960358 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" podUID="d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:47:44.076420 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:44.076377 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" podUID="0f0c53ac-dd7d-4ee0-b707-51857ccf2724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:47:44.205428 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:44.205381 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" podUID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 06:47:44.857376 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:44.834606 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" podUID="80db6c09-1940-4274-a21c-57ab7dd6738c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:47:44.917531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:44.917480 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" podUID="4199dace-66ce-4982-9ff6-629345b077ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:47:44.918168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:44.918147 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:47:45.017390 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:47:45.014491 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp" Oct 30 06:48:03.702614 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:03.702531 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:48:17.685392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:17.685352 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-d5q7w"] Oct 30 06:48:22.693429 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:22.693366 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" podUID="09a12623-f17f-4bdd-9198-ea086b422922" containerName="payload-processing" containerID="cri-o://8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747" gracePeriod=25 Oct 30 06:48:23.154046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.154021 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:48:23.363373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.361799 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-plugins-config-volume\") pod \"09a12623-f17f-4bdd-9198-ea086b422922\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " Oct 30 06:48:23.363373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.362031 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-routing-overlay-volume\") pod \"09a12623-f17f-4bdd-9198-ea086b422922\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " Oct 30 06:48:23.363373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.362129 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-kube-api-access-mb2sn\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-kube-api-access-mb2sn\") pod \"09a12623-f17f-4bdd-9198-ea086b422922\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " Oct 30 06:48:23.363373 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.362223 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-provider-credentials\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-provider-credentials\") pod \"09a12623-f17f-4bdd-9198-ea086b422922\" (UID: \"09a12623-f17f-4bdd-9198-ea086b422922\") " Oct 30 06:48:23.383387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.372509 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-routing-overlay-volume" pod "09a12623-f17f-4bdd-9198-ea086b422922" (UID: "09a12623-f17f-4bdd-9198-ea086b422922"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:48:23.399971 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.393576 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-plugins-config-volume" pod "09a12623-f17f-4bdd-9198-ea086b422922" (UID: "09a12623-f17f-4bdd-9198-ea086b422922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:48:23.415616 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.415564 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-kube-api-access-mb2sn" pod "09a12623-f17f-4bdd-9198-ea086b422922" (UID: "09a12623-f17f-4bdd-9198-ea086b422922"). InnerVolumeSpecName "kube-api-access-mb2sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:48:23.426778 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.426745 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-provider-credentials" pod "09a12623-f17f-4bdd-9198-ea086b422922" (UID: "09a12623-f17f-4bdd-9198-ea086b422922"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:48:23.433973 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.433938 4431 generic.go:356] "Generic (PLEG): container finished" podID="09a12623-f17f-4bdd-9198-ea086b422922" containerID="8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747" exitCode=0 Oct 30 06:48:23.434128 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.434060 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" event={"ID":"09a12623-f17f-4bdd-9198-ea086b422922","Type":"ContainerDied","Data":"8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747"} Oct 30 06:48:23.434128 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.434102 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" event={"ID":"09a12623-f17f-4bdd-9198-ea086b422922","Type":"ContainerDied","Data":"8dd315fd78b8b1645929628b0d4f0228ff2c624a69f1795f8ac4fcbce2c02723"} Oct 30 06:48:23.434128 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.434123 4431 scope.go:122] "RemoveContainer" containerID="8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747" Oct 30 06:48:23.434327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.434252 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-578778c84f-d5q7w" Oct 30 06:48:23.447864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.447846 4431 scope.go:122] "RemoveContainer" containerID="8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747" Oct 30 06:48:23.448153 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:23.448131 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747\": container with ID starting with 8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747 not found: ID does not exist" containerID="8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747" Oct 30 06:48:23.448226 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.448164 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747"} err="failed to get container status \"8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747\": rpc error: code = NotFound desc = could not find container \"8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747\": container with ID starting with 8411c0ae5bbb94ed3019b9750c6d220f4573c4c5834624e05a9442d7053f0747 not found: ID does not exist" Oct 30 06:48:23.463953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.463928 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:48:23.463953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.463952 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb2sn\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-kube-api-access-mb2sn\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:48:23.464075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.463962 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09a12623-f17f-4bdd-9198-ea086b422922-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:48:23.464075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.463970 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a12623-f17f-4bdd-9198-ea086b422922-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:48:23.487881 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.487851 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-d5q7w"] Oct 30 06:48:23.491086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:23.491063 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-578778c84f-d5q7w"] Oct 30 06:48:25.377620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:25.377573 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a12623-f17f-4bdd-9198-ea086b422922" path="/var/lib/kubelet/pods/09a12623-f17f-4bdd-9198-ea086b422922/volumes" Oct 30 06:48:26.201284 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.201254 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz"] Oct 30 06:48:26.201985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.201964 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerName="istio-proxy" Oct 30 06:48:26.201985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.201982 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerName="istio-proxy" Oct 30 06:48:26.201985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.201994 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" Oct 30 06:48:26.202183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.201998 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" Oct 30 06:48:26.202183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202050 4431 cpu_manager.go:414] "containerMap: removing container" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" Oct 30 06:48:26.202183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202061 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" Oct 30 06:48:26.202183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202088 4431 cpu_manager.go:414] "containerMap: removing container" podUID="09a12623-f17f-4bdd-9198-ea086b422922" containerName="payload-processing" Oct 30 06:48:26.202183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202095 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a12623-f17f-4bdd-9198-ea086b422922" containerName="payload-processing" Oct 30 06:48:26.202183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202123 4431 cpu_manager.go:414] "containerMap: removing container" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerName="istio-proxy" Oct 30 06:48:26.202183 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202132 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerName="istio-proxy" Oct 30 06:48:26.202435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202317 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a12623-f17f-4bdd-9198-ea086b422922" containerName="payload-processing" Oct 30 06:48:26.202435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202334 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ee2fcd-1dfb-47d6-b7aa-395160feb05a" containerName="istio-proxy" Oct 30 06:48:26.202435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202352 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a2e0884-320a-4cf8-ba2a-55d22ff54988" containerName="istio-proxy" Oct 30 06:48:26.202435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202359 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e912eea-45ff-47e0-b15b-165a912ce731" containerName="istio-proxy" Oct 30 06:48:26.202435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.202371 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="aebc7419-b24e-42f5-9107-fe3143d88963" containerName="istio-proxy" Oct 30 06:48:26.208094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.208067 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.222601 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.222578 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"agc-cred-18cbd8b523a4\"" Oct 30 06:48:26.224165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.224139 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz"] Oct 30 06:48:26.230730 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.230694 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d8df2\"" Oct 30 06:48:26.230730 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.230698 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"payload-processing-external-model-e2e--17d17c8c-dockercfg-ljq66\"" Oct 30 06:48:26.234632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.234609 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"openshift-service-ca.crt\"" Oct 30 06:48:26.234755 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.234642 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"routing-overlay\"" Oct 30 06:48:26.234879 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.234866 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d8df2\"/\"kube-root-ca.crt\"" Oct 30 06:48:26.287362 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.287298 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.287489 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.287389 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.287540 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.287522 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntpnk\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-kube-api-access-ntpnk\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.287691 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.287676 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.400325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.398262 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.400325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.399495 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.400325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.399546 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.400325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.399658 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntpnk\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-kube-api-access-ntpnk\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.405198 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.405166 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.405991 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.405967 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.424469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.424440 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntpnk\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-kube-api-access-ntpnk\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.448345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.441851 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.528686 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.528590 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:26.847620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:26.847592 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz"] Oct 30 06:48:26.848239 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:48:26.848207 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde29935c_f0a7_41e4_acce_423f1920ed05.slice/crio-f480655f97ec8c9913551abe24541fc787b13305cd49bd3f58500e05fe0ad766 WatchSource:0}: Error finding container f480655f97ec8c9913551abe24541fc787b13305cd49bd3f58500e05fe0ad766: Status 404 returned error can't find the container with id f480655f97ec8c9913551abe24541fc787b13305cd49bd3f58500e05fe0ad766 Oct 30 06:48:27.476788 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:27.476754 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" event={"ID":"de29935c-f0a7-41e4-acce-423f1920ed05","Type":"ContainerStarted","Data":"d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865"} Oct 30 06:48:27.477235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:27.476814 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:27.477235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:27.476835 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" event={"ID":"de29935c-f0a7-41e4-acce-423f1920ed05","Type":"ContainerStarted","Data":"f480655f97ec8c9913551abe24541fc787b13305cd49bd3f58500e05fe0ad766"} Oct 30 06:48:27.518997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:27.518946 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" podStartSLOduration=1.518929062 podStartE2EDuration="1.518929062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:27.518901966 +0000 UTC m=+3316.325423715" watchObservedRunningTime="2026-10-30 06:48:27.518929062 +0000 UTC m=+3316.325450812" Oct 30 06:48:28.667326 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:28.667278 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2d8df2/agc-cred-18cbd8b523a4: secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:28.667884 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:28.667861 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz: secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:28.668036 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:28.668024 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials podName:de29935c-f0a7-41e4-acce-423f1920ed05 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:29.168002387 +0000 UTC m=+3317.974524133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" (UID: "de29935c-f0a7-41e4-acce-423f1920ed05") : secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:29.114804 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:29.113372 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz"] Oct 30 06:48:29.178052 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:29.178023 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2d8df2/agc-cred-18cbd8b523a4: secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:29.178052 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:29.178047 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz: secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:29.178272 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:29.178099 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials podName:de29935c-f0a7-41e4-acce-423f1920ed05 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:30.178083235 +0000 UTC m=+3318.984604975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" (UID: "de29935c-f0a7-41e4-acce-423f1920ed05") : secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:30.195669 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:30.195631 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2d8df2/agc-cred-18cbd8b523a4: secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:30.195669 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:30.195661 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz: secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:30.196114 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:30.195719 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials podName:de29935c-f0a7-41e4-acce-423f1920ed05 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:32.195704854 +0000 UTC m=+3321.002226593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" (UID: "de29935c-f0a7-41e4-acce-423f1920ed05") : secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:32.220801 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:32.220769 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2d8df2/agc-cred-18cbd8b523a4: secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:32.220801 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:32.220795 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz: secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:32.221236 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:32.220861 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials podName:de29935c-f0a7-41e4-acce-423f1920ed05 nodeName:}" failed. No retries permitted until 2026-10-30 06:48:36.220847183 +0000 UTC m=+3325.027368909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" (UID: "de29935c-f0a7-41e4-acce-423f1920ed05") : secret "agc-cred-18cbd8b523a4" not found Oct 30 06:48:33.701742 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:33.701702 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:48:34.503011 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.502950 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" podUID="de29935c-f0a7-41e4-acce-423f1920ed05" containerName="payload-processing" containerID="cri-o://d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865" gracePeriod=25 Oct 30 06:48:34.771318 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.771247 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:34.864385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.864357 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials\") pod \"de29935c-f0a7-41e4-acce-423f1920ed05\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " Oct 30 06:48:34.864385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.864386 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-kube-api-access-ntpnk\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-kube-api-access-ntpnk\") pod \"de29935c-f0a7-41e4-acce-423f1920ed05\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " Oct 30 06:48:34.864664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.864490 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-routing-overlay-volume\") pod \"de29935c-f0a7-41e4-acce-423f1920ed05\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " Oct 30 06:48:34.864664 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.864598 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-plugins-config-volume\") pod \"de29935c-f0a7-41e4-acce-423f1920ed05\" (UID: \"de29935c-f0a7-41e4-acce-423f1920ed05\") " Oct 30 06:48:34.864902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.864869 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-routing-overlay-volume" pod "de29935c-f0a7-41e4-acce-423f1920ed05" (UID: "de29935c-f0a7-41e4-acce-423f1920ed05"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:48:34.865051 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.864966 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-plugins-config-volume" pod "de29935c-f0a7-41e4-acce-423f1920ed05" (UID: "de29935c-f0a7-41e4-acce-423f1920ed05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:48:34.865206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.865186 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-routing-overlay-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:48:34.865263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.865211 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de29935c-f0a7-41e4-acce-423f1920ed05-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:48:34.866536 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.866514 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials" pod "de29935c-f0a7-41e4-acce-423f1920ed05" (UID: "de29935c-f0a7-41e4-acce-423f1920ed05"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:48:34.866642 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.866579 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-kube-api-access-ntpnk" pod "de29935c-f0a7-41e4-acce-423f1920ed05" (UID: "de29935c-f0a7-41e4-acce-423f1920ed05"). InnerVolumeSpecName "kube-api-access-ntpnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:48:34.966920 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.966890 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-provider-credentials\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:48:34.966920 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:34.966918 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntpnk\" (UniqueName: \"kubernetes.io/projected/de29935c-f0a7-41e4-acce-423f1920ed05-kube-api-access-ntpnk\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:48:35.544438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.544383 4431 generic.go:356] "Generic (PLEG): container finished" podID="de29935c-f0a7-41e4-acce-423f1920ed05" containerID="d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865" exitCode=0 Oct 30 06:48:35.544438 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.544436 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" event={"ID":"de29935c-f0a7-41e4-acce-423f1920ed05","Type":"ContainerDied","Data":"d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865"} Oct 30 06:48:35.544654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.544485 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" event={"ID":"de29935c-f0a7-41e4-acce-423f1920ed05","Type":"ContainerDied","Data":"f480655f97ec8c9913551abe24541fc787b13305cd49bd3f58500e05fe0ad766"} Oct 30 06:48:35.544654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.544508 4431 scope.go:122] "RemoveContainer" containerID="d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865" Oct 30 06:48:35.544654 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.544439 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz" Oct 30 06:48:35.554665 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.554636 4431 scope.go:122] "RemoveContainer" containerID="d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865" Oct 30 06:48:35.554938 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:48:35.554918 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865\": container with ID starting with d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865 not found: ID does not exist" containerID="d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865" Oct 30 06:48:35.554981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.554947 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865"} err="failed to get container status \"d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865\": rpc error: code = NotFound desc = could not find container \"d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865\": container with ID starting with d8c85a2faef54c5923f4988177e7682fe41cadd3bb7d44a1bbe2d2c0cdc63865 not found: ID does not exist" Oct 30 06:48:35.596412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.596388 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz"] Oct 30 06:48:35.608410 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.608342 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d8df2/payload-processing-external-model-e2e-extmod-b-2d8df2-86b69shwz"] Oct 30 06:48:35.993042 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.993012 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl"] Oct 30 06:48:35.993700 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.993686 4431 cpu_manager.go:414] "containerMap: removing container" podUID="de29935c-f0a7-41e4-acce-423f1920ed05" containerName="payload-processing" Oct 30 06:48:35.993758 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.993700 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de29935c-f0a7-41e4-acce-423f1920ed05" containerName="payload-processing" Oct 30 06:48:35.993803 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.993793 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="de29935c-f0a7-41e4-acce-423f1920ed05" containerName="payload-processing" Oct 30 06:48:35.999151 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:35.999122 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.021347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.016078 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b22a13-serving-cert\"" Oct 30 06:48:36.021347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.021087 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl"] Oct 30 06:48:36.089063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.089025 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.089238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.089179 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-metrics-certs\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.089238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.089226 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.089355 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.089343 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lpz\" (UniqueName: \"kubernetes.io/projected/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-kube-api-access-w4lpz\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.190391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.190356 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lpz\" (UniqueName: \"kubernetes.io/projected/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-kube-api-access-w4lpz\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.190561 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.190541 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.190669 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.190654 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-metrics-certs\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.190726 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.190702 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.191379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.191354 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.193138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.193117 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-metrics-certs\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.193248 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.193169 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-maas-api-tls\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.213033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.213011 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lpz\" (UniqueName: \"kubernetes.io/projected/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-kube-api-access-w4lpz\") pod \"maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.353378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.343598 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:36.817732 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:36.817706 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl"] Oct 30 06:48:36.820102 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:48:36.820068 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9fb6fa_fbde_4cc5_b43d_8856815fb3cb.slice/crio-0e348a75471cc9b4940669db2e290384e9bba747d2153b64c75de8e99a6a69cf WatchSource:0}: Error finding container 0e348a75471cc9b4940669db2e290384e9bba747d2153b64c75de8e99a6a69cf: Status 404 returned error can't find the container with id 0e348a75471cc9b4940669db2e290384e9bba747d2153b64c75de8e99a6a69cf Oct 30 06:48:37.338492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:37.338454 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de29935c-f0a7-41e4-acce-423f1920ed05" path="/var/lib/kubelet/pods/de29935c-f0a7-41e4-acce-423f1920ed05/volumes" Oct 30 06:48:37.575116 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:37.575070 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" event={"ID":"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb","Type":"ContainerStarted","Data":"07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356"} Oct 30 06:48:37.575364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:37.575180 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" event={"ID":"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb","Type":"ContainerStarted","Data":"0e348a75471cc9b4940669db2e290384e9bba747d2153b64c75de8e99a6a69cf"} Oct 30 06:48:37.575756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:37.575734 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:48:37.618423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:37.618378 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" podStartSLOduration=2.321138639 podStartE2EDuration="2.618364824s" totalImagesPullingTime="297.226185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:37.617176908 +0000 UTC m=+3326.423698669" watchObservedRunningTime="2026-10-30 06:48:37.618364824 +0000 UTC m=+3326.424886614" Oct 30 06:48:43.588455 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:48:43.588423 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:49:03.109040 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.108995 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t"] Oct 30 06:49:03.114904 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.114877 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.130391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.130364 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t"] Oct 30 06:49:03.137997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.137972 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b66aad\"/\"default-dockercfg-bpj6t\"" Oct 30 06:49:03.138164 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.138100 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b66aad\"/\"openshift-service-ca.crt\"" Oct 30 06:49:03.143398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.143362 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b66aad\"/\"test-model-b66aad-kserve-self-signed-certs\"" Oct 30 06:49:03.143532 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.143418 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b66aad\"/\"kube-root-ca.crt\"" Oct 30 06:49:03.187492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.187464 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-dshm\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.187634 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.187507 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-home\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.187686 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.187663 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a2216c-9007-4a4e-99b7-0601e1d2694d-tls-certs\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.187733 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.187712 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-model-cache\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.187801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.187732 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98mjt\" (UniqueName: \"kubernetes.io/projected/99a2216c-9007-4a4e-99b7-0601e1d2694d-kube-api-access-98mjt\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.187899 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.187881 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-tmp-dir\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.289422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.289384 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-home\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.289602 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.289490 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a2216c-9007-4a4e-99b7-0601e1d2694d-tls-certs\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.289602 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.289531 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-model-cache\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.289602 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.289557 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98mjt\" (UniqueName: \"kubernetes.io/projected/99a2216c-9007-4a4e-99b7-0601e1d2694d-kube-api-access-98mjt\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.289783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.289635 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-tmp-dir\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.289783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.289770 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-home\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.289897 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.289845 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-dshm\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.289959 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.289892 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-model-cache\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.290017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.289974 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-tmp-dir\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.291917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.291895 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-dshm\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.292085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.292071 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a2216c-9007-4a4e-99b7-0601e1d2694d-tls-certs\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.312591 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.312564 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98mjt\" (UniqueName: \"kubernetes.io/projected/99a2216c-9007-4a4e-99b7-0601e1d2694d-kube-api-access-98mjt\") pod \"test-model-b66aad-kserve-784c8cfd55-c9p4t\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.432662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.432575 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:03.702608 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:49:03.702576 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:49:03.795896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.795867 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t"] Oct 30 06:49:03.796775 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:49:03.796751 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99a2216c_9007_4a4e_99b7_0601e1d2694d.slice/crio-a80e4f8cea57009786a01d56ff98ede5e0368b8551cd73ef8cc725c7bed4fc79 WatchSource:0}: Error finding container a80e4f8cea57009786a01d56ff98ede5e0368b8551cd73ef8cc725c7bed4fc79: Status 404 returned error can't find the container with id a80e4f8cea57009786a01d56ff98ede5e0368b8551cd73ef8cc725c7bed4fc79 Oct 30 06:49:03.836666 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:03.836641 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" event={"ID":"99a2216c-9007-4a4e-99b7-0601e1d2694d","Type":"ContainerStarted","Data":"a80e4f8cea57009786a01d56ff98ede5e0368b8551cd73ef8cc725c7bed4fc79"} Oct 30 06:49:04.896353 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:04.893485 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" event={"ID":"99a2216c-9007-4a4e-99b7-0601e1d2694d","Type":"ContainerStarted","Data":"a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3"} Oct 30 06:49:04.941764 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:04.941716 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" podStartSLOduration=1.9417007659999999 podStartE2EDuration="1.941700766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:49:04.93801417 +0000 UTC m=+3353.744535921" watchObservedRunningTime="2026-10-30 06:49:04.941700766 +0000 UTC m=+3353.748222566" Oct 30 06:49:05.433554 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:05.433523 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:10.861945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:10.861909 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr"] Oct 30 06:49:10.866357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:10.866339 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:10.884549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:10.884525 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr"] Oct 30 06:49:10.891887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:10.891861 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-89f101\"" Oct 30 06:49:11.044837 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.044793 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr\" (UID: \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:11.045392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.045353 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5thb\" (UniqueName: \"kubernetes.io/projected/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-kube-api-access-x5thb\") pod \"payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr\" (UID: \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:11.170411 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.170321 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5thb\" (UniqueName: \"kubernetes.io/projected/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-kube-api-access-x5thb\") pod \"payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr\" (UID: \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:11.171098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.171076 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr\" (UID: \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:11.178536 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.171995 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr\" (UID: \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:11.194948 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.194913 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5thb\" (UniqueName: \"kubernetes.io/projected/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-kube-api-access-x5thb\") pod \"payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr\" (UID: \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:11.481947 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.481917 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:11.786257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.786227 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr"] Oct 30 06:49:11.786879 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:49:11.786852 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c2c9b0_53dd_4f84_bc20_fc34e7aaf4bb.slice/crio-d50a96057901533512b68ff98db63d3f4e10dd6b117434aa87c829dd645bdfca WatchSource:0}: Error finding container d50a96057901533512b68ff98db63d3f4e10dd6b117434aa87c829dd645bdfca: Status 404 returned error can't find the container with id d50a96057901533512b68ff98db63d3f4e10dd6b117434aa87c829dd645bdfca Oct 30 06:49:11.978736 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.978698 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" event={"ID":"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb","Type":"ContainerStarted","Data":"fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49"} Oct 30 06:49:11.979101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.978765 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" event={"ID":"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb","Type":"ContainerStarted","Data":"d50a96057901533512b68ff98db63d3f4e10dd6b117434aa87c829dd645bdfca"} Oct 30 06:49:11.979101 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:11.978794 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:12.024618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:12.024572 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" podStartSLOduration=2.024559271 podStartE2EDuration="2.024559271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:49:12.019965026 +0000 UTC m=+3360.826486775" watchObservedRunningTime="2026-10-30 06:49:12.024559271 +0000 UTC m=+3360.831081034" Oct 30 06:49:13.434170 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:13.434130 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:13.446422 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:13.446397 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:14.001730 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:14.001699 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:49:22.988252 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:22.988222 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:49:33.750268 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:49:33.750213 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:49:42.467289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.467258 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx"] Oct 30 06:49:42.471931 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.471915 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:49:42.490560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.490531 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx"] Oct 30 06:49:42.498469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.498443 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a5f459\"" Oct 30 06:49:42.498469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.498461 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a5f459-dockercfg-cdjsm\"" Oct 30 06:49:42.626158 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.598335 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx\" (UID: \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:49:42.696739 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.696700 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8r9\" (UniqueName: \"kubernetes.io/projected/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-kube-api-access-8s8r9\") pod \"payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx\" (UID: \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:49:42.702391 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.702361 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2"] Oct 30 06:49:42.708285 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.708267 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:42.716563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.712521 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2"] Oct 30 06:49:42.799566 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.799535 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8r9\" (UniqueName: \"kubernetes.io/projected/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-kube-api-access-8s8r9\") pod \"payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx\" (UID: \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:49:42.799734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.799595 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918cfec5-83fd-469d-8a53-63990ff9376a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2\" (UID: \"918cfec5-83fd-469d-8a53-63990ff9376a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:42.799734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.799677 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjwlv\" (UniqueName: \"kubernetes.io/projected/918cfec5-83fd-469d-8a53-63990ff9376a-kube-api-access-gjwlv\") pod \"payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2\" (UID: \"918cfec5-83fd-469d-8a53-63990ff9376a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:42.799834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.799756 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx\" (UID: \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:49:42.800482 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.800460 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx\" (UID: \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:49:42.823657 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.823633 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8r9\" (UniqueName: \"kubernetes.io/projected/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-kube-api-access-8s8r9\") pod \"payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx\" (UID: \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:49:42.900982 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.900946 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918cfec5-83fd-469d-8a53-63990ff9376a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2\" (UID: \"918cfec5-83fd-469d-8a53-63990ff9376a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:42.901120 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.901077 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjwlv\" (UniqueName: \"kubernetes.io/projected/918cfec5-83fd-469d-8a53-63990ff9376a-kube-api-access-gjwlv\") pod \"payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2\" (UID: \"918cfec5-83fd-469d-8a53-63990ff9376a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:42.901722 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.901702 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918cfec5-83fd-469d-8a53-63990ff9376a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2\" (UID: \"918cfec5-83fd-469d-8a53-63990ff9376a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:42.924225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:42.924203 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjwlv\" (UniqueName: \"kubernetes.io/projected/918cfec5-83fd-469d-8a53-63990ff9376a-kube-api-access-gjwlv\") pod \"payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2\" (UID: \"918cfec5-83fd-469d-8a53-63990ff9376a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:43.037994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:43.037922 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:43.086075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:43.086039 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:49:43.343775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:43.343753 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2"] Oct 30 06:49:43.344707 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:49:43.344682 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918cfec5_83fd_469d_8a53_63990ff9376a.slice/crio-123f4728bfd8b6f696ccb98aba12f8ce68cdfda2461ef2a5b09e09b402d6646a WatchSource:0}: Error finding container 123f4728bfd8b6f696ccb98aba12f8ce68cdfda2461ef2a5b09e09b402d6646a: Status 404 returned error can't find the container with id 123f4728bfd8b6f696ccb98aba12f8ce68cdfda2461ef2a5b09e09b402d6646a Oct 30 06:49:43.418886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:43.418859 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx"] Oct 30 06:49:43.420346 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:49:43.420286 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7376eddd_dd2f_4c7a_af89_7c9fb70abf70.slice/crio-7982f3eac5729b6b3a6dfb86670e3e66e1a19941d24121c2d603c5d3e2c7f684 WatchSource:0}: Error finding container 7982f3eac5729b6b3a6dfb86670e3e66e1a19941d24121c2d603c5d3e2c7f684: Status 404 returned error can't find the container with id 7982f3eac5729b6b3a6dfb86670e3e66e1a19941d24121c2d603c5d3e2c7f684 Oct 30 06:49:44.224090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:44.224056 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" event={"ID":"918cfec5-83fd-469d-8a53-63990ff9376a","Type":"ContainerStarted","Data":"a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9"} Oct 30 06:49:44.224586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:44.224114 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" event={"ID":"918cfec5-83fd-469d-8a53-63990ff9376a","Type":"ContainerStarted","Data":"123f4728bfd8b6f696ccb98aba12f8ce68cdfda2461ef2a5b09e09b402d6646a"} Oct 30 06:49:44.224586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:44.224148 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:44.225873 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:44.225846 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" event={"ID":"7376eddd-dd2f-4c7a-af89-7c9fb70abf70","Type":"ContainerStarted","Data":"ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968"} Oct 30 06:49:44.225981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:44.225898 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" event={"ID":"7376eddd-dd2f-4c7a-af89-7c9fb70abf70","Type":"ContainerStarted","Data":"7982f3eac5729b6b3a6dfb86670e3e66e1a19941d24121c2d603c5d3e2c7f684"} Oct 30 06:49:44.225981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:44.225925 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:49:44.265589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:44.265536 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" podStartSLOduration=2.265519111 podStartE2EDuration="2.265519111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:49:44.263432269 +0000 UTC m=+3393.069954032" watchObservedRunningTime="2026-10-30 06:49:44.265519111 +0000 UTC m=+3393.072040858" Oct 30 06:49:44.304113 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:44.304064 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" podStartSLOduration=2.304050208 podStartE2EDuration="2.304050208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:49:44.302658613 +0000 UTC m=+3393.109180365" watchObservedRunningTime="2026-10-30 06:49:44.304050208 +0000 UTC m=+3393.110571955" Oct 30 06:49:55.234878 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:55.234849 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:49:55.235370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:49:55.234914 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:50:03.702846 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:50:03.702814 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:50:33.702840 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:50:33.702801 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:50:34.339369 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.336578 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c"] Oct 30 06:50:34.402364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.402025 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c"] Oct 30 06:50:34.402364 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.402176 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.429579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.429547 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a5f459-serving-cert\"" Oct 30 06:50:34.552577 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.543223 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcm6n\" (UniqueName: \"kubernetes.io/projected/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-kube-api-access-qcm6n\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.552955 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.552921 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-metrics-certs\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.553392 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.553283 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.553640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.553622 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-maas-api-tls\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.654896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.654815 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.655044 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.654955 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-maas-api-tls\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.655118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.655107 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcm6n\" (UniqueName: \"kubernetes.io/projected/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-kube-api-access-qcm6n\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.655157 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.655127 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-metrics-certs\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.655491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.655458 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.657731 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.657699 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-metrics-certs\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.657885 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.657834 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-maas-api-tls\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.678344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.678316 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcm6n\" (UniqueName: \"kubernetes.io/projected/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-kube-api-access-qcm6n\") pod \"maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:34.777499 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:34.777469 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:35.087667 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:35.087641 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c"] Oct 30 06:50:35.088881 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:50:35.088852 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdd9574_f2ab_449e_916e_74fd2f3cda2d.slice/crio-09d5da8e23ff74776f391a8281d6d794adeda68afcba2ea1b57f340e807319f3 WatchSource:0}: Error finding container 09d5da8e23ff74776f391a8281d6d794adeda68afcba2ea1b57f340e807319f3: Status 404 returned error can't find the container with id 09d5da8e23ff74776f391a8281d6d794adeda68afcba2ea1b57f340e807319f3 Oct 30 06:50:35.641900 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:35.641857 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" event={"ID":"3bdd9574-f2ab-449e-916e-74fd2f3cda2d","Type":"ContainerStarted","Data":"f4544dde7924ef435a1c381a7681abd90f4990de06c84676c9b5019c51773c2b"} Oct 30 06:50:35.642041 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:35.641916 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" event={"ID":"3bdd9574-f2ab-449e-916e-74fd2f3cda2d","Type":"ContainerStarted","Data":"09d5da8e23ff74776f391a8281d6d794adeda68afcba2ea1b57f340e807319f3"} Oct 30 06:50:35.642848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:35.642829 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:35.687253 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:35.687209 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" podStartSLOduration=1.272459561 podStartE2EDuration="1.687195933s" totalImagesPullingTime="414.736372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:50:35.686092634 +0000 UTC m=+3444.492614382" watchObservedRunningTime="2026-10-30 06:50:35.687195933 +0000 UTC m=+3444.493717680" Oct 30 06:50:40.605459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.605415 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp"] Oct 30 06:50:40.615018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.614983 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:50:40.660328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.651581 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-be48b4\"" Oct 30 06:50:40.660328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.651718 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-be48b4-dockercfg-nh9zn\"" Oct 30 06:50:40.697168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.697125 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp"] Oct 30 06:50:40.706001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.705779 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq9fl\" (UniqueName: \"kubernetes.io/projected/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-kube-api-access-fq9fl\") pod \"payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp\" (UID: \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:50:40.706439 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.706417 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp\" (UID: \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:50:40.807805 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.807744 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9fl\" (UniqueName: \"kubernetes.io/projected/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-kube-api-access-fq9fl\") pod \"payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp\" (UID: \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:50:40.808201 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.808182 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp\" (UID: \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:50:40.809317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.809261 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp\" (UID: \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:50:40.836319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:40.836277 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9fl\" (UniqueName: \"kubernetes.io/projected/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-kube-api-access-fq9fl\") pod \"payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp\" (UID: \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:50:41.006185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:41.006153 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:50:41.311350 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:41.311316 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp"] Oct 30 06:50:41.312432 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:50:41.312405 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ae7da1_de6f_4171_abcd_60c8ee6c9bdc.slice/crio-0ac59e3fa9d6ff8bf6b2e5fcdee0a38bb58ee7ff11357c42b62e776576890bd8 WatchSource:0}: Error finding container 0ac59e3fa9d6ff8bf6b2e5fcdee0a38bb58ee7ff11357c42b62e776576890bd8: Status 404 returned error can't find the container with id 0ac59e3fa9d6ff8bf6b2e5fcdee0a38bb58ee7ff11357c42b62e776576890bd8 Oct 30 06:50:41.698601 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:41.698566 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" event={"ID":"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc","Type":"ContainerStarted","Data":"ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab"} Oct 30 06:50:41.699066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:41.698627 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:50:41.699066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:41.698649 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" event={"ID":"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc","Type":"ContainerStarted","Data":"0ac59e3fa9d6ff8bf6b2e5fcdee0a38bb58ee7ff11357c42b62e776576890bd8"} Oct 30 06:50:41.741818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:41.741773 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" podStartSLOduration=1.7417555930000002 podStartE2EDuration="1.741755593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:50:41.738589165 +0000 UTC m=+3450.545110937" watchObservedRunningTime="2026-10-30 06:50:41.741755593 +0000 UTC m=+3450.548277342" Oct 30 06:50:42.662873 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:42.662841 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:50:52.707543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:50:52.707511 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:51:03.706846 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:51:03.706766 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:51:33.702839 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:51:33.702777 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:51:34.217288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.217257 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt"] Oct 30 06:51:34.222102 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.222077 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.240901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.240874 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt"] Oct 30 06:51:34.249665 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.249636 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b22a13\"/\"default-dockercfg-2hlz4\"" Oct 30 06:51:34.249798 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.249766 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b22a13\"/\"test-model-b22a13-kserve-self-signed-certs\"" Oct 30 06:51:34.319771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.319734 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b22a13\"/\"openshift-service-ca.crt\"" Oct 30 06:51:34.319771 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.319744 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b22a13\"/\"kube-root-ca.crt\"" Oct 30 06:51:34.335154 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.335059 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-home\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.335482 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.335464 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-tmp-dir\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.335826 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.335781 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6829b5b1-5091-4483-be7f-dd73b6a4428e-tls-certs\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.336121 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.336089 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-model-cache\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.336212 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.336191 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrhxk\" (UniqueName: \"kubernetes.io/projected/6829b5b1-5091-4483-be7f-dd73b6a4428e-kube-api-access-mrhxk\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.336433 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.336409 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-dshm\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.438149 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.438118 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-model-cache\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.438352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.438156 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrhxk\" (UniqueName: \"kubernetes.io/projected/6829b5b1-5091-4483-be7f-dd73b6a4428e-kube-api-access-mrhxk\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.438352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.438229 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-dshm\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.438352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.438268 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-home\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.438352 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.438316 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-tmp-dir\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.438573 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.438365 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6829b5b1-5091-4483-be7f-dd73b6a4428e-tls-certs\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.438727 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.438699 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-home\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.438813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.438777 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-tmp-dir\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.438949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.438922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-model-cache\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.440624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.440604 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-dshm\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.440879 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.440860 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6829b5b1-5091-4483-be7f-dd73b6a4428e-tls-certs\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.461045 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.461023 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrhxk\" (UniqueName: \"kubernetes.io/projected/6829b5b1-5091-4483-be7f-dd73b6a4428e-kube-api-access-mrhxk\") pod \"test-model-b22a13-kserve-d7896b77-lk7mt\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.540800 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.540728 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:34.963594 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:34.963559 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt"] Oct 30 06:51:34.964249 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:51:34.964227 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6829b5b1_5091_4483_be7f_dd73b6a4428e.slice/crio-39db005f22655a6aa8b3db14c5310542d872d14bae72bf25c39c891a93b3eb17 WatchSource:0}: Error finding container 39db005f22655a6aa8b3db14c5310542d872d14bae72bf25c39c891a93b3eb17: Status 404 returned error can't find the container with id 39db005f22655a6aa8b3db14c5310542d872d14bae72bf25c39c891a93b3eb17 Oct 30 06:51:35.164811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:35.164775 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" event={"ID":"6829b5b1-5091-4483-be7f-dd73b6a4428e","Type":"ContainerStarted","Data":"b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29"} Oct 30 06:51:35.164990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:35.164820 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" event={"ID":"6829b5b1-5091-4483-be7f-dd73b6a4428e","Type":"ContainerStarted","Data":"39db005f22655a6aa8b3db14c5310542d872d14bae72bf25c39c891a93b3eb17"} Oct 30 06:51:35.545325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:35.544410 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:44.541281 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:44.541246 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:44.554335 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:44.554295 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:51:44.600614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:44.600555 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" podStartSLOduration=10.600537967 podStartE2EDuration="10.600537967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:51:35.219650499 +0000 UTC m=+3504.026172249" watchObservedRunningTime="2026-10-30 06:51:44.600537967 +0000 UTC m=+3513.407059716" Oct 30 06:51:45.245852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:51:45.245818 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:52:03.780586 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:52:03.780542 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:52:23.353092 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:23.353060 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2"] Oct 30 06:52:28.354105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:28.354068 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" podUID="a9edcc58-67e7-4486-a23d-6893a284a2fd" containerName="payload-pre-processing" containerID="cri-o://eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412" gracePeriod=25 Oct 30 06:52:31.804656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:31.804615 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" podUID="a9edcc58-67e7-4486-a23d-6893a284a2fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:52:33.702020 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:52:33.701984 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:52:41.805194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:41.805153 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" podUID="a9edcc58-67e7-4486-a23d-6893a284a2fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:52:48.395497 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:52:48.395465 4431 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9edcc58_67e7_4486_a23d_6893a284a2fd.slice/crio-eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:52:48.692866 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.692841 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:52:48.697691 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.697668 4431 generic.go:356] "Generic (PLEG): container finished" podID="a9edcc58-67e7-4486-a23d-6893a284a2fd" containerID="eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412" exitCode=0 Oct 30 06:52:48.697821 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.697708 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" Oct 30 06:52:48.697821 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.697712 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" event={"ID":"a9edcc58-67e7-4486-a23d-6893a284a2fd","Type":"ContainerDied","Data":"eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412"} Oct 30 06:52:48.697958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.697863 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2" event={"ID":"a9edcc58-67e7-4486-a23d-6893a284a2fd","Type":"ContainerDied","Data":"4935b43bda4ce14d513dc78cd747dbab4c8ea6c61bb1d90fa7feb8d0d0240ead"} Oct 30 06:52:48.697958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.697908 4431 scope.go:122] "RemoveContainer" containerID="eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412" Oct 30 06:52:48.708444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.708422 4431 scope.go:122] "RemoveContainer" containerID="eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412" Oct 30 06:52:48.708688 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:52:48.708667 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412\": container with ID starting with eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412 not found: ID does not exist" containerID="eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412" Oct 30 06:52:48.708742 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.708695 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412"} err="failed to get container status \"eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412\": rpc error: code = NotFound desc = could not find container \"eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412\": container with ID starting with eda19667815d0fbf82ce7b1c2957c3efa37ad9a18dde5c0e673d3a4da7cc5412 not found: ID does not exist" Oct 30 06:52:48.783401 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.783362 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9edcc58-67e7-4486-a23d-6893a284a2fd-kube-api-access-r78kh\" (UniqueName: \"kubernetes.io/projected/a9edcc58-67e7-4486-a23d-6893a284a2fd-kube-api-access-r78kh\") pod \"a9edcc58-67e7-4486-a23d-6893a284a2fd\" (UID: \"a9edcc58-67e7-4486-a23d-6893a284a2fd\") " Oct 30 06:52:48.783651 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.783630 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9edcc58-67e7-4486-a23d-6893a284a2fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9edcc58-67e7-4486-a23d-6893a284a2fd-plugins-config-volume\") pod \"a9edcc58-67e7-4486-a23d-6893a284a2fd\" (UID: \"a9edcc58-67e7-4486-a23d-6893a284a2fd\") " Oct 30 06:52:48.784094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.784060 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9edcc58-67e7-4486-a23d-6893a284a2fd-plugins-config-volume" pod "a9edcc58-67e7-4486-a23d-6893a284a2fd" (UID: "a9edcc58-67e7-4486-a23d-6893a284a2fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:52:48.784418 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.784395 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9edcc58-67e7-4486-a23d-6893a284a2fd-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:52:48.785608 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.785581 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9edcc58-67e7-4486-a23d-6893a284a2fd-kube-api-access-r78kh" pod "a9edcc58-67e7-4486-a23d-6893a284a2fd" (UID: "a9edcc58-67e7-4486-a23d-6893a284a2fd"). InnerVolumeSpecName "kube-api-access-r78kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:52:48.885847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:48.885820 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r78kh\" (UniqueName: \"kubernetes.io/projected/a9edcc58-67e7-4486-a23d-6893a284a2fd-kube-api-access-r78kh\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:52:49.046014 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:49.045986 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2"] Oct 30 06:52:49.057542 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:49.057519 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2d2923-7f66c948d7-jn9z2"] Oct 30 06:52:49.339760 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:52:49.339670 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9edcc58-67e7-4486-a23d-6893a284a2fd" path="/var/lib/kubelet/pods/a9edcc58-67e7-4486-a23d-6893a284a2fd/volumes" Oct 30 06:53:03.702033 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:53:03.702000 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:53:20.600858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:20.600822 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t"] Oct 30 06:53:31.697327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:31.692217 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l"] Oct 30 06:53:31.713541 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:31.713498 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb"] Oct 30 06:53:33.724789 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:53:33.724756 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:53:35.622880 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.622834 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" podUID="99a2216c-9007-4a4e-99b7-0601e1d2694d" containerName="main" containerID="cri-o://a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3" gracePeriod=45 Oct 30 06:53:35.901708 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.901685 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:53:35.981089 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.981056 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99a2216c-9007-4a4e-99b7-0601e1d2694d-tls-certs\" (UniqueName: \"kubernetes.io/secret/99a2216c-9007-4a4e-99b7-0601e1d2694d-tls-certs\") pod \"99a2216c-9007-4a4e-99b7-0601e1d2694d\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " Oct 30 06:53:35.981261 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.981148 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-dshm\") pod \"99a2216c-9007-4a4e-99b7-0601e1d2694d\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " Oct 30 06:53:35.981261 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.981224 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a2216c-9007-4a4e-99b7-0601e1d2694d-kube-api-access-98mjt\" (UniqueName: \"kubernetes.io/projected/99a2216c-9007-4a4e-99b7-0601e1d2694d-kube-api-access-98mjt\") pod \"99a2216c-9007-4a4e-99b7-0601e1d2694d\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " Oct 30 06:53:35.981261 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.981252 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-model-cache\") pod \"99a2216c-9007-4a4e-99b7-0601e1d2694d\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " Oct 30 06:53:35.981504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.981394 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-tmp-dir\") pod \"99a2216c-9007-4a4e-99b7-0601e1d2694d\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " Oct 30 06:53:35.981504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.981471 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-home\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-home\") pod \"99a2216c-9007-4a4e-99b7-0601e1d2694d\" (UID: \"99a2216c-9007-4a4e-99b7-0601e1d2694d\") " Oct 30 06:53:35.982019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.981658 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-tmp-dir" pod "99a2216c-9007-4a4e-99b7-0601e1d2694d" (UID: "99a2216c-9007-4a4e-99b7-0601e1d2694d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:35.982019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.981830 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-home" pod "99a2216c-9007-4a4e-99b7-0601e1d2694d" (UID: "99a2216c-9007-4a4e-99b7-0601e1d2694d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:35.982019 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.982004 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-model-cache" pod "99a2216c-9007-4a4e-99b7-0601e1d2694d" (UID: "99a2216c-9007-4a4e-99b7-0601e1d2694d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:35.982204 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.982187 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:35.982265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.982214 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:35.982265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.982227 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:35.983969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.983942 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-dshm" pod "99a2216c-9007-4a4e-99b7-0601e1d2694d" (UID: "99a2216c-9007-4a4e-99b7-0601e1d2694d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:35.984067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.983996 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a2216c-9007-4a4e-99b7-0601e1d2694d-kube-api-access-98mjt" pod "99a2216c-9007-4a4e-99b7-0601e1d2694d" (UID: "99a2216c-9007-4a4e-99b7-0601e1d2694d"). InnerVolumeSpecName "kube-api-access-98mjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:35.984136 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:35.984066 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a2216c-9007-4a4e-99b7-0601e1d2694d-tls-certs" pod "99a2216c-9007-4a4e-99b7-0601e1d2694d" (UID: "99a2216c-9007-4a4e-99b7-0601e1d2694d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:36.084001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.083971 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a2216c-9007-4a4e-99b7-0601e1d2694d-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:36.084115 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.084003 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98mjt\" (UniqueName: \"kubernetes.io/projected/99a2216c-9007-4a4e-99b7-0601e1d2694d-kube-api-access-98mjt\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:36.084115 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.084015 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a2216c-9007-4a4e-99b7-0601e1d2694d-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:36.238078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.238045 4431 generic.go:356] "Generic (PLEG): container finished" podID="99a2216c-9007-4a4e-99b7-0601e1d2694d" containerID="a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3" exitCode=0 Oct 30 06:53:36.238258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.238093 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" Oct 30 06:53:36.238258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.238123 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" event={"ID":"99a2216c-9007-4a4e-99b7-0601e1d2694d","Type":"ContainerDied","Data":"a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3"} Oct 30 06:53:36.238258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.238167 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t" event={"ID":"99a2216c-9007-4a4e-99b7-0601e1d2694d","Type":"ContainerDied","Data":"a80e4f8cea57009786a01d56ff98ede5e0368b8551cd73ef8cc725c7bed4fc79"} Oct 30 06:53:36.238258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.238184 4431 scope.go:122] "RemoveContainer" containerID="a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3" Oct 30 06:53:36.249958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.249944 4431 scope.go:122] "RemoveContainer" containerID="a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3" Oct 30 06:53:36.250198 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:53:36.250179 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3\": container with ID starting with a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3 not found: ID does not exist" containerID="a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3" Oct 30 06:53:36.250287 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.250204 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3"} err="failed to get container status \"a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3\": rpc error: code = NotFound desc = could not find container \"a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3\": container with ID starting with a7e486d088f5b4a2afe921023e981ec0c3b59c5ffed35609202c35ac8846bbc3 not found: ID does not exist" Oct 30 06:53:36.291312 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.291272 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t"] Oct 30 06:53:36.295371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.295348 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b66aad/test-model-b66aad-kserve-784c8cfd55-c9p4t"] Oct 30 06:53:36.714263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:36.714212 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" podUID="22c3879a-73ba-46f3-b878-5cd4d76c286e" containerName="payload-pre-processing" containerID="cri-o://44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9" gracePeriod=25 Oct 30 06:53:37.206741 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:37.206704 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" podUID="6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" containerName="payload-processing" containerID="cri-o://77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab" gracePeriod=25 Oct 30 06:53:37.339319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:37.339275 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a2216c-9007-4a4e-99b7-0601e1d2694d" path="/var/lib/kubelet/pods/99a2216c-9007-4a4e-99b7-0601e1d2694d/volumes" Oct 30 06:53:38.952727 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:38.952694 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8"] Oct 30 06:53:38.953502 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:38.953469 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerName="istio-proxy" containerID="cri-o://e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e" gracePeriod=30 Oct 30 06:53:44.252647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.252621 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:53:44.310460 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.310428 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-certs\") pod \"c4613cd6-89a5-49aa-aca4-566a8605cce7\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " Oct 30 06:53:44.310622 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.310605 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-data\") pod \"c4613cd6-89a5-49aa-aca4-566a8605cce7\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " Oct 30 06:53:44.310683 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.310653 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-token\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-token\") pod \"c4613cd6-89a5-49aa-aca4-566a8605cce7\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " Oct 30 06:53:44.310743 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.310693 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-podinfo\") pod \"c4613cd6-89a5-49aa-aca4-566a8605cce7\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " Oct 30 06:53:44.310743 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.310714 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-certs" pod "c4613cd6-89a5-49aa-aca4-566a8605cce7" (UID: "c4613cd6-89a5-49aa-aca4-566a8605cce7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.310853 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.310777 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4613cd6-89a5-49aa-aca4-566a8605cce7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4613cd6-89a5-49aa-aca4-566a8605cce7-istiod-ca-cert\") pod \"c4613cd6-89a5-49aa-aca4-566a8605cce7\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " Oct 30 06:53:44.310967 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.310946 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-envoy\") pod \"c4613cd6-89a5-49aa-aca4-566a8605cce7\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " Oct 30 06:53:44.311062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.310995 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-socket\") pod \"c4613cd6-89a5-49aa-aca4-566a8605cce7\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " Oct 30 06:53:44.311062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.311036 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-kube-api-access-4w4m4\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-kube-api-access-4w4m4\") pod \"c4613cd6-89a5-49aa-aca4-566a8605cce7\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " Oct 30 06:53:44.311224 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.311073 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-credential-socket\") pod \"c4613cd6-89a5-49aa-aca4-566a8605cce7\" (UID: \"c4613cd6-89a5-49aa-aca4-566a8605cce7\") " Oct 30 06:53:44.311408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.311297 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-socket" pod "c4613cd6-89a5-49aa-aca4-566a8605cce7" (UID: "c4613cd6-89a5-49aa-aca4-566a8605cce7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.311699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.311651 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4613cd6-89a5-49aa-aca4-566a8605cce7-istiod-ca-cert" pod "c4613cd6-89a5-49aa-aca4-566a8605cce7" (UID: "c4613cd6-89a5-49aa-aca4-566a8605cce7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:44.311860 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.311836 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.311976 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.311866 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.312104 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.312086 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-data" pod "c4613cd6-89a5-49aa-aca4-566a8605cce7" (UID: "c4613cd6-89a5-49aa-aca4-566a8605cce7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.312591 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.312562 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-credential-socket" pod "c4613cd6-89a5-49aa-aca4-566a8605cce7" (UID: "c4613cd6-89a5-49aa-aca4-566a8605cce7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.317551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.317433 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-podinfo" pod "c4613cd6-89a5-49aa-aca4-566a8605cce7" (UID: "c4613cd6-89a5-49aa-aca4-566a8605cce7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:53:44.317551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.317431 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-envoy" pod "c4613cd6-89a5-49aa-aca4-566a8605cce7" (UID: "c4613cd6-89a5-49aa-aca4-566a8605cce7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:44.317551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.317491 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-kube-api-access-4w4m4" pod "c4613cd6-89a5-49aa-aca4-566a8605cce7" (UID: "c4613cd6-89a5-49aa-aca4-566a8605cce7"). InnerVolumeSpecName "kube-api-access-4w4m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:44.317551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.317526 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-token" pod "c4613cd6-89a5-49aa-aca4-566a8605cce7" (UID: "c4613cd6-89a5-49aa-aca4-566a8605cce7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:44.325757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.325732 4431 generic.go:356] "Generic (PLEG): container finished" podID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerID="e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e" exitCode=0 Oct 30 06:53:44.325852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.325794 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" Oct 30 06:53:44.325919 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.325787 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" event={"ID":"c4613cd6-89a5-49aa-aca4-566a8605cce7","Type":"ContainerDied","Data":"e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e"} Oct 30 06:53:44.325978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.325951 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8" event={"ID":"c4613cd6-89a5-49aa-aca4-566a8605cce7","Type":"ContainerDied","Data":"621fa711c9137f51fde216a455bf5a4cc68ba71c3f8c5c327891485fdd1d1676"} Oct 30 06:53:44.326033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.325988 4431 scope.go:122] "RemoveContainer" containerID="e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e" Oct 30 06:53:44.355141 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.355120 4431 scope.go:122] "RemoveContainer" containerID="e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e" Oct 30 06:53:44.355452 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:53:44.355433 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e\": container with ID starting with e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e not found: ID does not exist" containerID="e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e" Oct 30 06:53:44.355529 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.355460 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e"} err="failed to get container status \"e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e\": rpc error: code = NotFound desc = could not find container \"e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e\": container with ID starting with e597bef6d36b25946e0fca424be43f7273502941844cb431bce383587db71a5e not found: ID does not exist" Oct 30 06:53:44.396396 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.396367 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8"] Oct 30 06:53:44.397647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.397625 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-bqpx8"] Oct 30 06:53:44.412932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.412905 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.412932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.412931 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4m4\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-kube-api-access-4w4m4\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.413078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.412945 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.413078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.412958 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.413078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.412971 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.413078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.412984 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4613cd6-89a5-49aa-aca4-566a8605cce7-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.413078 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.412996 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4613cd6-89a5-49aa-aca4-566a8605cce7-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:44.751259 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.751225 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" podUID="22c3879a-73ba-46f3-b878-5cd4d76c286e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:53:44.751259 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:44.751253 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" podUID="6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:53:45.339486 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:45.339452 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" path="/var/lib/kubelet/pods/c4613cd6-89a5-49aa-aca4-566a8605cce7/volumes" Oct 30 06:53:54.751589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:54.751494 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" podUID="22c3879a-73ba-46f3-b878-5cd4d76c286e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:53:54.751589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:54.751530 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" podUID="6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:53:57.008703 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.008678 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:53:57.190778 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.190745 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c3879a-73ba-46f3-b878-5cd4d76c286e-kube-api-access-28h67\" (UniqueName: \"kubernetes.io/projected/22c3879a-73ba-46f3-b878-5cd4d76c286e-kube-api-access-28h67\") pod \"22c3879a-73ba-46f3-b878-5cd4d76c286e\" (UID: \"22c3879a-73ba-46f3-b878-5cd4d76c286e\") " Oct 30 06:53:57.190945 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.190856 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c3879a-73ba-46f3-b878-5cd4d76c286e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3879a-73ba-46f3-b878-5cd4d76c286e-plugins-config-volume\") pod \"22c3879a-73ba-46f3-b878-5cd4d76c286e\" (UID: \"22c3879a-73ba-46f3-b878-5cd4d76c286e\") " Oct 30 06:53:57.191222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.191198 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c3879a-73ba-46f3-b878-5cd4d76c286e-plugins-config-volume" pod "22c3879a-73ba-46f3-b878-5cd4d76c286e" (UID: "22c3879a-73ba-46f3-b878-5cd4d76c286e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:57.191362 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.191346 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3879a-73ba-46f3-b878-5cd4d76c286e-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:57.192834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.192810 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c3879a-73ba-46f3-b878-5cd4d76c286e-kube-api-access-28h67" pod "22c3879a-73ba-46f3-b878-5cd4d76c286e" (UID: "22c3879a-73ba-46f3-b878-5cd4d76c286e"). InnerVolumeSpecName "kube-api-access-28h67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:57.292336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.292263 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28h67\" (UniqueName: \"kubernetes.io/projected/22c3879a-73ba-46f3-b878-5cd4d76c286e-kube-api-access-28h67\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:57.455861 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.455830 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:53:57.462923 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.462897 4431 generic.go:356] "Generic (PLEG): container finished" podID="6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" containerID="77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab" exitCode=0 Oct 30 06:53:57.463046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.462940 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" Oct 30 06:53:57.463046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.462981 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" event={"ID":"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3","Type":"ContainerDied","Data":"77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab"} Oct 30 06:53:57.463046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.463030 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l" event={"ID":"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3","Type":"ContainerDied","Data":"28e20f819226e12bca103b3ec08e97dafc4c05f8a584355ec4e4f71ad3b3165d"} Oct 30 06:53:57.463202 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.463056 4431 scope.go:122] "RemoveContainer" containerID="77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab" Oct 30 06:53:57.464524 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.464501 4431 generic.go:356] "Generic (PLEG): container finished" podID="22c3879a-73ba-46f3-b878-5cd4d76c286e" containerID="44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9" exitCode=0 Oct 30 06:53:57.464648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.464557 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" event={"ID":"22c3879a-73ba-46f3-b878-5cd4d76c286e","Type":"ContainerDied","Data":"44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9"} Oct 30 06:53:57.464648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.464563 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" Oct 30 06:53:57.464648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.464592 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb" event={"ID":"22c3879a-73ba-46f3-b878-5cd4d76c286e","Type":"ContainerDied","Data":"c0616f342a419c8fa8d9f5c74ba4fd4e0b012d327b6110555318d1c9ab34cc70"} Oct 30 06:53:57.477361 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.477335 4431 scope.go:122] "RemoveContainer" containerID="77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab" Oct 30 06:53:57.477657 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:53:57.477636 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab\": container with ID starting with 77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab not found: ID does not exist" containerID="77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab" Oct 30 06:53:57.477756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.477664 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab"} err="failed to get container status \"77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab\": rpc error: code = NotFound desc = could not find container \"77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab\": container with ID starting with 77f919e8c49950ff2bda1e0b82cc0c5d58505ecf69570ff35964b9638d25d7ab not found: ID does not exist" Oct 30 06:53:57.477756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.477684 4431 scope.go:122] "RemoveContainer" containerID="44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9" Oct 30 06:53:57.486676 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.486657 4431 scope.go:122] "RemoveContainer" containerID="44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9" Oct 30 06:53:57.486920 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:53:57.486902 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9\": container with ID starting with 44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9 not found: ID does not exist" containerID="44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9" Oct 30 06:53:57.486972 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.486927 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9"} err="failed to get container status \"44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9\": rpc error: code = NotFound desc = could not find container \"44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9\": container with ID starting with 44fae7420d043b45af5285a523581211f8f456cbef1d7f41f486ab3b0f9d94a9 not found: ID does not exist" Oct 30 06:53:57.536386 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.536353 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb"] Oct 30 06:53:57.541781 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.541752 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c148a-5d48d58c47-jkzsb"] Oct 30 06:53:57.599876 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.599818 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-plugins-config-volume\") pod \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\" (UID: \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\") " Oct 30 06:53:57.599973 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.599934 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-kube-api-access-sqqbt\" (UniqueName: \"kubernetes.io/projected/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-kube-api-access-sqqbt\") pod \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\" (UID: \"6a60cbf4-4f6e-4bef-810a-6d72d11feaa3\") " Oct 30 06:53:57.600203 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.600182 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-plugins-config-volume" pod "6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" (UID: "6a60cbf4-4f6e-4bef-810a-6d72d11feaa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:57.600562 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.600545 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:57.601939 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.601919 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-kube-api-access-sqqbt" pod "6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" (UID: "6a60cbf4-4f6e-4bef-810a-6d72d11feaa3"). InnerVolumeSpecName "kube-api-access-sqqbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:57.702356 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.702292 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqqbt\" (UniqueName: \"kubernetes.io/projected/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3-kube-api-access-sqqbt\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:57.808372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.808344 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l"] Oct 30 06:53:57.819662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:57.819635 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c148a-69b6466f68-4557l"] Oct 30 06:53:58.345092 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.345057 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf"] Oct 30 06:53:58.345546 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.345275 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" containerID="cri-o://9d0e8c31511d1c4152624a3a1da8e60d578199daa43020cbbace6ead5d24f63c" gracePeriod=30 Oct 30 06:53:58.422263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.422224 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng"] Oct 30 06:53:58.422569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.422542 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" podUID="3c067b93-6503-40a6-9d51-bc566d49d33d" containerName="maas-api" containerID="cri-o://86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558" gracePeriod=30 Oct 30 06:53:58.822540 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.822504 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:53:58.949258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.921873 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c067b93-6503-40a6-9d51-bc566d49d33d-kube-api-access-2lvdx\" (UniqueName: \"kubernetes.io/projected/3c067b93-6503-40a6-9d51-bc566d49d33d-kube-api-access-2lvdx\") pod \"3c067b93-6503-40a6-9d51-bc566d49d33d\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " Oct 30 06:53:58.949258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.922184 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-metrics-certs\") pod \"3c067b93-6503-40a6-9d51-bc566d49d33d\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " Oct 30 06:53:58.949258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.922659 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-maas-api-tls\") pod \"3c067b93-6503-40a6-9d51-bc566d49d33d\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " Oct 30 06:53:58.949258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.922935 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c067b93-6503-40a6-9d51-bc566d49d33d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c067b93-6503-40a6-9d51-bc566d49d33d-openshift-service-ca\") pod \"3c067b93-6503-40a6-9d51-bc566d49d33d\" (UID: \"3c067b93-6503-40a6-9d51-bc566d49d33d\") " Oct 30 06:53:58.950702 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.950666 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c067b93-6503-40a6-9d51-bc566d49d33d-openshift-service-ca" pod "3c067b93-6503-40a6-9d51-bc566d49d33d" (UID: "3c067b93-6503-40a6-9d51-bc566d49d33d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:58.965756 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.965727 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c067b93-6503-40a6-9d51-bc566d49d33d-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:58.972561 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.970411 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-metrics-certs" pod "3c067b93-6503-40a6-9d51-bc566d49d33d" (UID: "3c067b93-6503-40a6-9d51-bc566d49d33d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:58.976323 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.973743 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-maas-api-tls" pod "3c067b93-6503-40a6-9d51-bc566d49d33d" (UID: "3c067b93-6503-40a6-9d51-bc566d49d33d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:58.980879 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:58.980836 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c067b93-6503-40a6-9d51-bc566d49d33d-kube-api-access-2lvdx" pod "3c067b93-6503-40a6-9d51-bc566d49d33d" (UID: "3c067b93-6503-40a6-9d51-bc566d49d33d"). InnerVolumeSpecName "kube-api-access-2lvdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:59.070418 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.070362 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:59.070418 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.070386 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lvdx\" (UniqueName: \"kubernetes.io/projected/3c067b93-6503-40a6-9d51-bc566d49d33d-kube-api-access-2lvdx\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:59.070418 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.070396 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c067b93-6503-40a6-9d51-bc566d49d33d-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:53:59.340621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.340545 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c3879a-73ba-46f3-b878-5cd4d76c286e" path="/var/lib/kubelet/pods/22c3879a-73ba-46f3-b878-5cd4d76c286e/volumes" Oct 30 06:53:59.341023 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.341009 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" path="/var/lib/kubelet/pods/6a60cbf4-4f6e-4bef-810a-6d72d11feaa3/volumes" Oct 30 06:53:59.341596 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.341578 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx"] Oct 30 06:53:59.342835 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342816 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3c067b93-6503-40a6-9d51-bc566d49d33d" containerName="maas-api" Oct 30 06:53:59.342886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342839 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c067b93-6503-40a6-9d51-bc566d49d33d" containerName="maas-api" Oct 30 06:53:59.342886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342872 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a9edcc58-67e7-4486-a23d-6893a284a2fd" containerName="payload-pre-processing" Oct 30 06:53:59.342886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342881 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9edcc58-67e7-4486-a23d-6893a284a2fd" containerName="payload-pre-processing" Oct 30 06:53:59.343007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342892 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" containerName="payload-processing" Oct 30 06:53:59.343007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342901 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" containerName="payload-processing" Oct 30 06:53:59.343007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342933 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerName="istio-proxy" Oct 30 06:53:59.343007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342941 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerName="istio-proxy" Oct 30 06:53:59.343007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342954 4431 cpu_manager.go:414] "containerMap: removing container" podUID="99a2216c-9007-4a4e-99b7-0601e1d2694d" containerName="main" Oct 30 06:53:59.343007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342961 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a2216c-9007-4a4e-99b7-0601e1d2694d" containerName="main" Oct 30 06:53:59.343007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342976 4431 cpu_manager.go:414] "containerMap: removing container" podUID="22c3879a-73ba-46f3-b878-5cd4d76c286e" containerName="payload-pre-processing" Oct 30 06:53:59.343007 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.342983 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c3879a-73ba-46f3-b878-5cd4d76c286e" containerName="payload-pre-processing" Oct 30 06:53:59.343268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.343140 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4613cd6-89a5-49aa-aca4-566a8605cce7" containerName="istio-proxy" Oct 30 06:53:59.343268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.343154 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a2216c-9007-4a4e-99b7-0601e1d2694d" containerName="main" Oct 30 06:53:59.343268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.343164 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c3879a-73ba-46f3-b878-5cd4d76c286e" containerName="payload-pre-processing" Oct 30 06:53:59.343268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.343184 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c067b93-6503-40a6-9d51-bc566d49d33d" containerName="maas-api" Oct 30 06:53:59.343268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.343196 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9edcc58-67e7-4486-a23d-6893a284a2fd" containerName="payload-pre-processing" Oct 30 06:53:59.343268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.343205 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a60cbf4-4f6e-4bef-810a-6d72d11feaa3" containerName="payload-processing" Oct 30 06:53:59.348844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.348826 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.358594 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.358572 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx"] Oct 30 06:53:59.365186 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.365166 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-eaee88-openshift-default-dockercfg-fmzvg\"" Oct 30 06:53:59.476003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.475964 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.476189 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.476085 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48caab37-f6bb-49eb-8ac9-1390b217079f-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.476189 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.476125 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.476610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.476584 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.476884 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.476825 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.477062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.476925 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.477062 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.477019 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9g9m\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-kube-api-access-r9g9m\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.477195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.477153 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.477263 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.477235 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.495038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.495008 4431 generic.go:356] "Generic (PLEG): container finished" podID="3c067b93-6503-40a6-9d51-bc566d49d33d" containerID="86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558" exitCode=0 Oct 30 06:53:59.495145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.495062 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" Oct 30 06:53:59.495145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.495093 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" event={"ID":"3c067b93-6503-40a6-9d51-bc566d49d33d","Type":"ContainerDied","Data":"86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558"} Oct 30 06:53:59.495222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.495146 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng" event={"ID":"3c067b93-6503-40a6-9d51-bc566d49d33d","Type":"ContainerDied","Data":"0c7d4d198f1108584417c20dd2ec49aff7e11e3af9c486b47f967e5e9aa4d280"} Oct 30 06:53:59.495222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.495172 4431 scope.go:122] "RemoveContainer" containerID="86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558" Oct 30 06:53:59.505699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.505683 4431 scope.go:122] "RemoveContainer" containerID="86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558" Oct 30 06:53:59.505964 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:53:59.505947 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558\": container with ID starting with 86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558 not found: ID does not exist" containerID="86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558" Oct 30 06:53:59.506012 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.505978 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558"} err="failed to get container status \"86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558\": rpc error: code = NotFound desc = could not find container \"86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558\": container with ID starting with 86280d1f66ce954877560735b1115843d484069ad8a0cbf4f55810534d0c0558 not found: ID does not exist" Oct 30 06:53:59.546718 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.546695 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng"] Oct 30 06:53:59.549974 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.549956 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c148a-76d9cc9cd7-48kng"] Oct 30 06:53:59.578551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.578528 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.578649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.578556 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9g9m\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-kube-api-access-r9g9m\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.578649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.578608 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.578783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.578649 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.578783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.578688 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.578783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.578738 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48caab37-f6bb-49eb-8ac9-1390b217079f-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.578783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.578758 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.578783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.578776 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.579053 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.578917 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.579053 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.579035 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.579191 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.579166 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.579292 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.579255 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.579447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.579395 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.579447 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.579427 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48caab37-f6bb-49eb-8ac9-1390b217079f-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.580988 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.580963 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.581222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.581203 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.601863 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.601812 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.601946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.601925 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9g9m\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-kube-api-access-r9g9m\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.670227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.670201 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:53:59.993334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.993290 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx"] Oct 30 06:53:59.994260 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:53:59.994227 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48caab37_f6bb_49eb_8ac9_1390b217079f.slice/crio-d5b3fce7f1f7282dafdc8d07d41b0bf1aeaeccf05ed5cf7cd490c1f24ea210c4 WatchSource:0}: Error finding container d5b3fce7f1f7282dafdc8d07d41b0bf1aeaeccf05ed5cf7cd490c1f24ea210c4: Status 404 returned error can't find the container with id d5b3fce7f1f7282dafdc8d07d41b0bf1aeaeccf05ed5cf7cd490c1f24ea210c4 Oct 30 06:53:59.997699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.997662 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:53:59.997801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.997759 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:53:59.997844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:53:59.997805 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:54:00.506561 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:00.506529 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" event={"ID":"48caab37-f6bb-49eb-8ac9-1390b217079f","Type":"ContainerStarted","Data":"15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed"} Oct 30 06:54:00.506962 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:00.506579 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" event={"ID":"48caab37-f6bb-49eb-8ac9-1390b217079f","Type":"ContainerStarted","Data":"d5b3fce7f1f7282dafdc8d07d41b0bf1aeaeccf05ed5cf7cd490c1f24ea210c4"} Oct 30 06:54:00.551770 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:00.551704 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" podStartSLOduration=1.551684438 podStartE2EDuration="1.551684438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:54:00.55150655 +0000 UTC m=+3649.358028297" watchObservedRunningTime="2026-10-30 06:54:00.551684438 +0000 UTC m=+3649.358206187" Oct 30 06:54:00.673326 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:00.670418 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:54:01.361340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:01.359475 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c067b93-6503-40a6-9d51-bc566d49d33d" path="/var/lib/kubelet/pods/3c067b93-6503-40a6-9d51-bc566d49d33d/volumes" Oct 30 06:54:01.685449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:01.685359 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:54:01.685449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:01.685419 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 06:54:02.586399 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:02.586362 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:54:02.586569 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:02.586435 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:54:02.676334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:02.675454 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:54:02.676334 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:02.675511 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 06:54:03.531586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.531545 4431 generic.go:356] "Generic (PLEG): container finished" podID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerID="9d0e8c31511d1c4152624a3a1da8e60d578199daa43020cbbace6ead5d24f63c" exitCode=0 Oct 30 06:54:03.531946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.531634 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" event={"ID":"f58609e0-908f-4c1a-93f8-a4cde0954868","Type":"ContainerDied","Data":"9d0e8c31511d1c4152624a3a1da8e60d578199daa43020cbbace6ead5d24f63c"} Oct 30 06:54:03.689775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.684407 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:54:03.689775 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.684464 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 06:54:03.705167 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.705142 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:54:03.712881 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:54:03.712848 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:54:03.841085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.841054 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-podinfo\") pod \"f58609e0-908f-4c1a-93f8-a4cde0954868\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " Oct 30 06:54:03.841423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.841409 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-token\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-token\") pod \"f58609e0-908f-4c1a-93f8-a4cde0954868\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " Oct 30 06:54:03.841469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.841454 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-kube-api-access-95nzp\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-kube-api-access-95nzp\") pod \"f58609e0-908f-4c1a-93f8-a4cde0954868\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " Oct 30 06:54:03.841513 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.841474 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-envoy\") pod \"f58609e0-908f-4c1a-93f8-a4cde0954868\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " Oct 30 06:54:03.841551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.841543 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-certs\") pod \"f58609e0-908f-4c1a-93f8-a4cde0954868\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " Oct 30 06:54:03.841587 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.841578 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-data\") pod \"f58609e0-908f-4c1a-93f8-a4cde0954868\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " Oct 30 06:54:03.841636 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.841624 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-socket\") pod \"f58609e0-908f-4c1a-93f8-a4cde0954868\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " Oct 30 06:54:03.841738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.841726 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58609e0-908f-4c1a-93f8-a4cde0954868-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58609e0-908f-4c1a-93f8-a4cde0954868-istiod-ca-cert\") pod \"f58609e0-908f-4c1a-93f8-a4cde0954868\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " Oct 30 06:54:03.841813 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.841783 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-credential-socket\") pod \"f58609e0-908f-4c1a-93f8-a4cde0954868\" (UID: \"f58609e0-908f-4c1a-93f8-a4cde0954868\") " Oct 30 06:54:03.842171 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.842145 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-certs" pod "f58609e0-908f-4c1a-93f8-a4cde0954868" (UID: "f58609e0-908f-4c1a-93f8-a4cde0954868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:03.842291 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.842263 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-socket" pod "f58609e0-908f-4c1a-93f8-a4cde0954868" (UID: "f58609e0-908f-4c1a-93f8-a4cde0954868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:03.842431 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.842408 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-data" pod "f58609e0-908f-4c1a-93f8-a4cde0954868" (UID: "f58609e0-908f-4c1a-93f8-a4cde0954868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:03.842497 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.842439 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.842778 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.842755 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-credential-socket" pod "f58609e0-908f-4c1a-93f8-a4cde0954868" (UID: "f58609e0-908f-4c1a-93f8-a4cde0954868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:03.842861 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.842794 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58609e0-908f-4c1a-93f8-a4cde0954868-istiod-ca-cert" pod "f58609e0-908f-4c1a-93f8-a4cde0954868" (UID: "f58609e0-908f-4c1a-93f8-a4cde0954868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:03.843526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.843504 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-podinfo" pod "f58609e0-908f-4c1a-93f8-a4cde0954868" (UID: "f58609e0-908f-4c1a-93f8-a4cde0954868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:54:03.844740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.844716 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-kube-api-access-95nzp" pod "f58609e0-908f-4c1a-93f8-a4cde0954868" (UID: "f58609e0-908f-4c1a-93f8-a4cde0954868"). InnerVolumeSpecName "kube-api-access-95nzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:03.844965 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.844940 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-token" pod "f58609e0-908f-4c1a-93f8-a4cde0954868" (UID: "f58609e0-908f-4c1a-93f8-a4cde0954868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:03.845008 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.844973 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-envoy" pod "f58609e0-908f-4c1a-93f8-a4cde0954868" (UID: "f58609e0-908f-4c1a-93f8-a4cde0954868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:03.944327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.944275 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.944487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.944341 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58609e0-908f-4c1a-93f8-a4cde0954868-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.944487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.944358 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.944487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.944367 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.944487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.944376 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.944487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.944385 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95nzp\" (UniqueName: \"kubernetes.io/projected/f58609e0-908f-4c1a-93f8-a4cde0954868-kube-api-access-95nzp\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.944487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.944393 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:03.944487 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:03.944401 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58609e0-908f-4c1a-93f8-a4cde0954868-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:04.540210 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:04.540181 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" Oct 30 06:54:04.540711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:04.540222 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf" event={"ID":"f58609e0-908f-4c1a-93f8-a4cde0954868","Type":"ContainerDied","Data":"15f1afe9c7d776ddae2a9f3c53bcdc39da5e1beebcf2edbd9424194e8ecd4aae"} Oct 30 06:54:04.540711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:04.540289 4431 scope.go:122] "RemoveContainer" containerID="9d0e8c31511d1c4152624a3a1da8e60d578199daa43020cbbace6ead5d24f63c" Oct 30 06:54:04.609993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:04.609963 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf"] Oct 30 06:54:04.614928 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:04.614902 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-dx7vf"] Oct 30 06:54:04.672619 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:04.672589 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:54:04.672847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:04.672821 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 06:54:05.339818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:05.339784 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" path="/var/lib/kubelet/pods/f58609e0-908f-4c1a-93f8-a4cde0954868/volumes" Oct 30 06:54:05.684626 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:05.684498 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:54:05.684626 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:05.684556 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 06:54:06.650469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:06.646424 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr"] Oct 30 06:54:06.650469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:06.646769 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" containerID="cri-o://737860eeb84ff3f1d6deb4f6f02711278d9993305023543f4f3578d79c29ed85" gracePeriod=30 Oct 30 06:54:06.670759 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:06.670732 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:54:06.670892 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:06.670780 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 06:54:07.676526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:07.676496 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:54:07.676981 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:07.676965 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:54:07.677617 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:07.677596 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:54:11.942476 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.942453 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:54:11.983281 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983254 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-podinfo\") pod \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " Oct 30 06:54:11.983454 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983420 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istiod-ca-cert\") pod \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " Oct 30 06:54:11.983522 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983453 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-certs\") pod \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " Oct 30 06:54:11.983679 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983653 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-credential-socket\") pod \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " Oct 30 06:54:11.983821 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983730 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-kube-api-access-lt8qq\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-kube-api-access-lt8qq\") pod \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " Oct 30 06:54:11.983821 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983741 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-certs" pod "9773d54c-d0fa-441b-b9b7-fe731d630d3a" (UID: "9773d54c-d0fa-441b-b9b7-fe731d630d3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:11.983949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983820 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-envoy\") pod \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " Oct 30 06:54:11.983949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983856 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istiod-ca-cert" pod "9773d54c-d0fa-441b-b9b7-fe731d630d3a" (UID: "9773d54c-d0fa-441b-b9b7-fe731d630d3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:11.983949 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983869 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-data\") pod \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " Oct 30 06:54:11.984118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.983946 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-token\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-token\") pod \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " Oct 30 06:54:11.984118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.984014 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-socket\") pod \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\" (UID: \"9773d54c-d0fa-441b-b9b7-fe731d630d3a\") " Oct 30 06:54:11.984241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.984179 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-data" pod "9773d54c-d0fa-441b-b9b7-fe731d630d3a" (UID: "9773d54c-d0fa-441b-b9b7-fe731d630d3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:11.984241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.984197 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-credential-socket" pod "9773d54c-d0fa-441b-b9b7-fe731d630d3a" (UID: "9773d54c-d0fa-441b-b9b7-fe731d630d3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:11.984501 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.984476 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-socket" pod "9773d54c-d0fa-441b-b9b7-fe731d630d3a" (UID: "9773d54c-d0fa-441b-b9b7-fe731d630d3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:11.984755 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.984731 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:11.984859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.984762 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:11.984859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.984777 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:11.984859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.984791 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:11.984859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.984804 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:11.985723 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.985688 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-podinfo" pod "9773d54c-d0fa-441b-b9b7-fe731d630d3a" (UID: "9773d54c-d0fa-441b-b9b7-fe731d630d3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:54:11.985979 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.985954 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-kube-api-access-lt8qq" pod "9773d54c-d0fa-441b-b9b7-fe731d630d3a" (UID: "9773d54c-d0fa-441b-b9b7-fe731d630d3a"). InnerVolumeSpecName "kube-api-access-lt8qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:11.986556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.986531 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-envoy" pod "9773d54c-d0fa-441b-b9b7-fe731d630d3a" (UID: "9773d54c-d0fa-441b-b9b7-fe731d630d3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:11.986619 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:11.986565 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-token" pod "9773d54c-d0fa-441b-b9b7-fe731d630d3a" (UID: "9773d54c-d0fa-441b-b9b7-fe731d630d3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:12.093378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.092800 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8qq\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-kube-api-access-lt8qq\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:12.093378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.092853 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:12.093378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.092873 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:12.093378 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.092885 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9773d54c-d0fa-441b-b9b7-fe731d630d3a-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:12.538354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.538315 4431 scope.go:122] "RemoveContainer" containerID="8da47eee2ac4486c631fef1c07b0dd1c0aac445035677a776160a8db06417b29" Oct 30 06:54:12.549055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.549035 4431 scope.go:122] "RemoveContainer" containerID="737860eeb84ff3f1d6deb4f6f02711278d9993305023543f4f3578d79c29ed85" Oct 30 06:54:12.597512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.597482 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" event={"ID":"9773d54c-d0fa-441b-b9b7-fe731d630d3a","Type":"ContainerDied","Data":"737860eeb84ff3f1d6deb4f6f02711278d9993305023543f4f3578d79c29ed85"} Oct 30 06:54:12.597618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.597561 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" event={"ID":"9773d54c-d0fa-441b-b9b7-fe731d630d3a","Type":"ContainerDied","Data":"68b5cb3f3a505568b723ebf30e3e3c3383defb4adc5bd41b719ecc6efa8b02ae"} Oct 30 06:54:12.597618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.597584 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr" Oct 30 06:54:12.680948 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.680919 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr"] Oct 30 06:54:12.683978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:12.683955 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1c148a-openshift-default-6b558c7dd7-rh2fr"] Oct 30 06:54:13.339146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:13.339114 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" path="/var/lib/kubelet/pods/9773d54c-d0fa-441b-b9b7-fe731d630d3a/volumes" Oct 30 06:54:18.750005 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:18.749968 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr"] Oct 30 06:54:18.750538 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:18.750215 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" containerID="cri-o://edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c" gracePeriod=30 Oct 30 06:54:18.761961 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:18.761929 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd"] Oct 30 06:54:18.762197 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:18.762169 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" containerID="cri-o://491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9" gracePeriod=30 Oct 30 06:54:18.765494 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:18.765470 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk"] Oct 30 06:54:18.765686 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:18.765662 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" containerID="cri-o://ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5" gracePeriod=30 Oct 30 06:54:18.774134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:18.774111 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg"] Oct 30 06:54:18.774292 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:18.774271 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" containerID="cri-o://38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb" gracePeriod=30 Oct 30 06:54:22.787225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:22.784619 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5mhqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:54:22.787225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:22.784699 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:54:24.094259 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.094223 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:54:24.200004 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.199917 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:54:24.204643 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.204615 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istiod-ca-cert\") pod \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " Oct 30 06:54:24.204768 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.204734 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-kube-api-access-kblq2\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-kube-api-access-kblq2\") pod \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " Oct 30 06:54:24.204842 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.204790 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-data\") pod \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " Oct 30 06:54:24.204842 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.204827 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-credential-socket\") pod \"8edcace5-3093-479b-9b0f-3d89251b214a\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " Oct 30 06:54:24.204948 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.204908 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-envoy\") pod \"8edcace5-3093-479b-9b0f-3d89251b214a\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " Oct 30 06:54:24.205011 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.204944 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-socket\") pod \"8edcace5-3093-479b-9b0f-3d89251b214a\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " Oct 30 06:54:24.205011 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.204989 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-envoy\") pod \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " Oct 30 06:54:24.205154 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205022 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-data\") pod \"8edcace5-3093-479b-9b0f-3d89251b214a\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " Oct 30 06:54:24.205154 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205102 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8edcace5-3093-479b-9b0f-3d89251b214a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edcace5-3093-479b-9b0f-3d89251b214a-istio-podinfo\") pod \"8edcace5-3093-479b-9b0f-3d89251b214a\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " Oct 30 06:54:24.205154 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205145 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-kube-api-access-zbtmd\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-kube-api-access-zbtmd\") pod \"8edcace5-3093-479b-9b0f-3d89251b214a\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " Oct 30 06:54:24.205330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205178 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-certs\") pod \"8edcace5-3093-479b-9b0f-3d89251b214a\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " Oct 30 06:54:24.205330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205215 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8edcace5-3093-479b-9b0f-3d89251b214a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edcace5-3093-479b-9b0f-3d89251b214a-istiod-ca-cert\") pod \"8edcace5-3093-479b-9b0f-3d89251b214a\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " Oct 30 06:54:24.205330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205312 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-istio-token\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-istio-token\") pod \"8edcace5-3093-479b-9b0f-3d89251b214a\" (UID: \"8edcace5-3093-479b-9b0f-3d89251b214a\") " Oct 30 06:54:24.205495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205352 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-certs\") pod \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " Oct 30 06:54:24.205495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205389 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-socket\") pod \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " Oct 30 06:54:24.205495 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205476 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-token\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-token\") pod \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " Oct 30 06:54:24.205662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205557 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-credential-socket\") pod \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " Oct 30 06:54:24.205662 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.205597 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-podinfo\") pod \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\" (UID: \"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632\") " Oct 30 06:54:24.206673 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.206137 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:54:24.206673 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.206523 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-socket" pod "8edcace5-3093-479b-9b0f-3d89251b214a" (UID: "8edcace5-3093-479b-9b0f-3d89251b214a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.208383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.207036 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-data" pod "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" (UID: "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.208383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.207285 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-credential-socket" pod "8edcace5-3093-479b-9b0f-3d89251b214a" (UID: "8edcace5-3093-479b-9b0f-3d89251b214a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.208383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.207612 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istiod-ca-cert" pod "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" (UID: "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:24.208383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.208020 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8edcace5-3093-479b-9b0f-3d89251b214a-istiod-ca-cert" pod "8edcace5-3093-479b-9b0f-3d89251b214a" (UID: "8edcace5-3093-479b-9b0f-3d89251b214a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:24.208383 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.208186 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-data" pod "8edcace5-3093-479b-9b0f-3d89251b214a" (UID: "8edcace5-3093-479b-9b0f-3d89251b214a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.208715 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.208676 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-socket" pod "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" (UID: "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.208788 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.208771 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-certs" pod "8edcace5-3093-479b-9b0f-3d89251b214a" (UID: "8edcace5-3093-479b-9b0f-3d89251b214a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.208966 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.208949 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-certs" pod "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" (UID: "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.210781 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.209541 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-envoy" pod "8edcace5-3093-479b-9b0f-3d89251b214a" (UID: "8edcace5-3093-479b-9b0f-3d89251b214a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.210781 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.209681 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-kube-api-access-kblq2" pod "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" (UID: "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632"). InnerVolumeSpecName "kube-api-access-kblq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.211004 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.210964 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-credential-socket" pod "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" (UID: "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.212824 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.212781 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-istio-token" pod "8edcace5-3093-479b-9b0f-3d89251b214a" (UID: "8edcace5-3093-479b-9b0f-3d89251b214a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.213888 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.213870 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:54:24.215437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.215400 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-podinfo" pod "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" (UID: "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:54:24.216235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.216214 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-kube-api-access-zbtmd" pod "8edcace5-3093-479b-9b0f-3d89251b214a" (UID: "8edcace5-3093-479b-9b0f-3d89251b214a"). InnerVolumeSpecName "kube-api-access-zbtmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.216461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.216379 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8edcace5-3093-479b-9b0f-3d89251b214a-istio-podinfo" pod "8edcace5-3093-479b-9b0f-3d89251b214a" (UID: "8edcace5-3093-479b-9b0f-3d89251b214a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:54:24.216461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.216389 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-envoy" pod "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" (UID: "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.217901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.217867 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-token" pod "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" (UID: "bd2b3564-82c5-4f7a-8b4e-9bdec23c5632"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.306859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.306771 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-podinfo\") pod \"f98ee5c8-0953-4edf-8207-3db4f00c709b\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " Oct 30 06:54:24.307126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.307105 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-certs\") pod \"f98ee5c8-0953-4edf-8207-3db4f00c709b\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " Oct 30 06:54:24.307257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.307242 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-token\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-token\") pod \"f98ee5c8-0953-4edf-8207-3db4f00c709b\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " Oct 30 06:54:24.307388 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.307373 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-certs\") pod \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " Oct 30 06:54:24.307529 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.307511 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-credential-socket\") pod \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " Oct 30 06:54:24.307663 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.307647 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-envoy\") pod \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " Oct 30 06:54:24.307786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.307771 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-podinfo\") pod \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " Oct 30 06:54:24.307922 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.307909 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98ee5c8-0953-4edf-8207-3db4f00c709b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98ee5c8-0953-4edf-8207-3db4f00c709b-istiod-ca-cert\") pod \"f98ee5c8-0953-4edf-8207-3db4f00c709b\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " Oct 30 06:54:24.308051 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.308036 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-socket\") pod \"f98ee5c8-0953-4edf-8207-3db4f00c709b\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " Oct 30 06:54:24.308848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.308807 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istiod-ca-cert\") pod \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " Oct 30 06:54:24.309041 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.309023 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-kube-api-access-72n7r\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-kube-api-access-72n7r\") pod \"f98ee5c8-0953-4edf-8207-3db4f00c709b\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " Oct 30 06:54:24.309176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.309161 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-socket\") pod \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " Oct 30 06:54:24.313694 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.313675 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-token\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-token\") pod \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " Oct 30 06:54:24.314295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.312040 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98ee5c8-0953-4edf-8207-3db4f00c709b-istiod-ca-cert" pod "f98ee5c8-0953-4edf-8207-3db4f00c709b" (UID: "f98ee5c8-0953-4edf-8207-3db4f00c709b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:24.314295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.312760 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istiod-ca-cert" pod "55c1bc35-ccc6-43d1-9694-494bba09ab8e" (UID: "55c1bc35-ccc6-43d1-9694-494bba09ab8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:24.314295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.312989 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-certs" pod "55c1bc35-ccc6-43d1-9694-494bba09ab8e" (UID: "55c1bc35-ccc6-43d1-9694-494bba09ab8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.314295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.313153 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-credential-socket" pod "55c1bc35-ccc6-43d1-9694-494bba09ab8e" (UID: "55c1bc35-ccc6-43d1-9694-494bba09ab8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.314295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.313335 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-certs" pod "f98ee5c8-0953-4edf-8207-3db4f00c709b" (UID: "f98ee5c8-0953-4edf-8207-3db4f00c709b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.314295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.313572 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-socket" pod "f98ee5c8-0953-4edf-8207-3db4f00c709b" (UID: "f98ee5c8-0953-4edf-8207-3db4f00c709b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.315990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.315950 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-data" pod "f98ee5c8-0953-4edf-8207-3db4f00c709b" (UID: "f98ee5c8-0953-4edf-8207-3db4f00c709b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.316163 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.313613 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-socket" pod "55c1bc35-ccc6-43d1-9694-494bba09ab8e" (UID: "55c1bc35-ccc6-43d1-9694-494bba09ab8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.316345 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.314151 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-data\") pod \"f98ee5c8-0953-4edf-8207-3db4f00c709b\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " Oct 30 06:54:24.316530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.316515 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-envoy\") pod \"f98ee5c8-0953-4edf-8207-3db4f00c709b\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " Oct 30 06:54:24.317098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.317080 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-data\") pod \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " Oct 30 06:54:24.317554 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.317434 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-data" pod "55c1bc35-ccc6-43d1-9694-494bba09ab8e" (UID: "55c1bc35-ccc6-43d1-9694-494bba09ab8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.317717 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.317700 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-credential-socket\") pod \"f98ee5c8-0953-4edf-8207-3db4f00c709b\" (UID: \"f98ee5c8-0953-4edf-8207-3db4f00c709b\") " Oct 30 06:54:24.318104 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.317988 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-credential-socket" pod "f98ee5c8-0953-4edf-8207-3db4f00c709b" (UID: "f98ee5c8-0953-4edf-8207-3db4f00c709b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.318521 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.318064 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-kube-api-access-lhw66\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-kube-api-access-lhw66\") pod \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\" (UID: \"55c1bc35-ccc6-43d1-9694-494bba09ab8e\") " Oct 30 06:54:24.319739 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.319722 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.319891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.319877 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322042 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322068 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322082 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98ee5c8-0953-4edf-8207-3db4f00c709b-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322097 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322111 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edcace5-3093-479b-9b0f-3d89251b214a-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322127 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322142 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbtmd\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-kube-api-access-zbtmd\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322160 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322175 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322189 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edcace5-3093-479b-9b0f-3d89251b214a-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322203 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322216 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edcace5-3093-479b-9b0f-3d89251b214a-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322230 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322243 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322260 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322273 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322288 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322318 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322334 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.322342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322348 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.323405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322364 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kblq2\" (UniqueName: \"kubernetes.io/projected/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-kube-api-access-kblq2\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.323405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322378 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.323405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322391 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.323405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322404 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edcace5-3093-479b-9b0f-3d89251b214a-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.323405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322417 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.323405 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.322431 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.329193 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.328997 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-kube-api-access-lhw66" pod "55c1bc35-ccc6-43d1-9694-494bba09ab8e" (UID: "55c1bc35-ccc6-43d1-9694-494bba09ab8e"). InnerVolumeSpecName "kube-api-access-lhw66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.329193 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.329049 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-token" pod "55c1bc35-ccc6-43d1-9694-494bba09ab8e" (UID: "55c1bc35-ccc6-43d1-9694-494bba09ab8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.329193 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.329161 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-envoy" pod "f98ee5c8-0953-4edf-8207-3db4f00c709b" (UID: "f98ee5c8-0953-4edf-8207-3db4f00c709b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.330580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.330463 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-podinfo" pod "55c1bc35-ccc6-43d1-9694-494bba09ab8e" (UID: "55c1bc35-ccc6-43d1-9694-494bba09ab8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:54:24.330580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.330478 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-kube-api-access-72n7r" pod "f98ee5c8-0953-4edf-8207-3db4f00c709b" (UID: "f98ee5c8-0953-4edf-8207-3db4f00c709b"). InnerVolumeSpecName "kube-api-access-72n7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.330580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.330526 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-token" pod "f98ee5c8-0953-4edf-8207-3db4f00c709b" (UID: "f98ee5c8-0953-4edf-8207-3db4f00c709b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:24.330580 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.330544 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-podinfo" pod "f98ee5c8-0953-4edf-8207-3db4f00c709b" (UID: "f98ee5c8-0953-4edf-8207-3db4f00c709b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:54:24.330845 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.330590 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-envoy" pod "55c1bc35-ccc6-43d1-9694-494bba09ab8e" (UID: "55c1bc35-ccc6-43d1-9694-494bba09ab8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:24.423997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.423933 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.423997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.423967 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.423997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.423981 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72n7r\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-kube-api-access-72n7r\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.423997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.423996 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.423997 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.424006 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.424337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.424015 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhw66\" (UniqueName: \"kubernetes.io/projected/55c1bc35-ccc6-43d1-9694-494bba09ab8e-kube-api-access-lhw66\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.424337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.424025 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.424337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.424033 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98ee5c8-0953-4edf-8207-3db4f00c709b-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:24.709100 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.709063 4431 generic.go:356] "Generic (PLEG): container finished" podID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerID="edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c" exitCode=0 Oct 30 06:54:24.709328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.709125 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" Oct 30 06:54:24.709328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.709149 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" event={"ID":"f98ee5c8-0953-4edf-8207-3db4f00c709b","Type":"ContainerDied","Data":"edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c"} Oct 30 06:54:24.709328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.709211 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr" event={"ID":"f98ee5c8-0953-4edf-8207-3db4f00c709b","Type":"ContainerDied","Data":"fd16d11b980471dc00c26b153d320f13ac7bcb0c9bde464f619642a54cd01a35"} Oct 30 06:54:24.709328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.709243 4431 scope.go:122] "RemoveContainer" containerID="edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c" Oct 30 06:54:24.710806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.710782 4431 generic.go:356] "Generic (PLEG): container finished" podID="8edcace5-3093-479b-9b0f-3d89251b214a" containerID="38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb" exitCode=0 Oct 30 06:54:24.710915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.710832 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" Oct 30 06:54:24.710915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.710861 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" event={"ID":"8edcace5-3093-479b-9b0f-3d89251b214a","Type":"ContainerDied","Data":"38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb"} Oct 30 06:54:24.711027 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.710914 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg" event={"ID":"8edcace5-3093-479b-9b0f-3d89251b214a","Type":"ContainerDied","Data":"b6d229f255e7753a2f845138e1a10d2b4a26b3756f2436c0899827a55e0ca527"} Oct 30 06:54:24.712542 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.712516 4431 generic.go:356] "Generic (PLEG): container finished" podID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerID="ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5" exitCode=0 Oct 30 06:54:24.712712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.712555 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" Oct 30 06:54:24.712712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.712586 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" event={"ID":"55c1bc35-ccc6-43d1-9694-494bba09ab8e","Type":"ContainerDied","Data":"ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5"} Oct 30 06:54:24.712712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.712645 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk" event={"ID":"55c1bc35-ccc6-43d1-9694-494bba09ab8e","Type":"ContainerDied","Data":"858f931c1c4230e8efb32d3a7cb5bd89162dc66d849dd18cd3e38162a5c88f8f"} Oct 30 06:54:24.714351 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.714329 4431 generic.go:356] "Generic (PLEG): container finished" podID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerID="491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9" exitCode=0 Oct 30 06:54:24.714469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.714390 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" Oct 30 06:54:24.714530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.714393 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" event={"ID":"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632","Type":"ContainerDied","Data":"491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9"} Oct 30 06:54:24.714530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.714513 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd" event={"ID":"bd2b3564-82c5-4f7a-8b4e-9bdec23c5632","Type":"ContainerDied","Data":"1838c877dca72d1ed7dd8d2cd4b7093f2e92720ccffc14efa84564aa19d62cca"} Oct 30 06:54:24.722265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.722243 4431 scope.go:122] "RemoveContainer" containerID="edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c" Oct 30 06:54:24.722619 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:54:24.722599 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c\": container with ID starting with edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c not found: ID does not exist" containerID="edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c" Oct 30 06:54:24.722710 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.722624 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c"} err="failed to get container status \"edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c\": rpc error: code = NotFound desc = could not find container \"edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c\": container with ID starting with edfad735207da631266cf5484985c57616e8ae115adc1b9b1f615839cbcdc12c not found: ID does not exist" Oct 30 06:54:24.722710 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.722642 4431 scope.go:122] "RemoveContainer" containerID="38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb" Oct 30 06:54:24.733247 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.733227 4431 scope.go:122] "RemoveContainer" containerID="38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb" Oct 30 06:54:24.733522 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:54:24.733504 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb\": container with ID starting with 38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb not found: ID does not exist" containerID="38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb" Oct 30 06:54:24.733610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.733532 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb"} err="failed to get container status \"38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb\": rpc error: code = NotFound desc = could not find container \"38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb\": container with ID starting with 38d3d18ecda0381ae625dc1f8d9a776326edd32d782a07047ccf20124b1d6dbb not found: ID does not exist" Oct 30 06:54:24.733610 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.733554 4431 scope.go:122] "RemoveContainer" containerID="ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5" Oct 30 06:54:24.742656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.742638 4431 scope.go:122] "RemoveContainer" containerID="ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5" Oct 30 06:54:24.742937 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:54:24.742919 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5\": container with ID starting with ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5 not found: ID does not exist" containerID="ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5" Oct 30 06:54:24.742985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.742943 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5"} err="failed to get container status \"ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5\": rpc error: code = NotFound desc = could not find container \"ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5\": container with ID starting with ec23a12db0e49e75afa7f6f02f790d22b947f3e2226ca753844b936d97c9deb5 not found: ID does not exist" Oct 30 06:54:24.742985 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.742956 4431 scope.go:122] "RemoveContainer" containerID="491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9" Oct 30 06:54:24.751814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.751794 4431 scope.go:122] "RemoveContainer" containerID="491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9" Oct 30 06:54:24.752059 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:54:24.752044 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9\": container with ID starting with 491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9 not found: ID does not exist" containerID="491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9" Oct 30 06:54:24.752119 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.752062 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9"} err="failed to get container status \"491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9\": rpc error: code = NotFound desc = could not find container \"491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9\": container with ID starting with 491fb874233e3e3d720d11573e4c6b389eb9cf5ab943cbd84960ac175eda2ad9 not found: ID does not exist" Oct 30 06:54:24.807126 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.807099 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd"] Oct 30 06:54:24.812357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.812335 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7t4cd"] Oct 30 06:54:24.846045 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.846020 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk"] Oct 30 06:54:24.851050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.851027 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmjhk"] Oct 30 06:54:24.891021 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.890977 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg"] Oct 30 06:54:24.896218 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.896170 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4gvmg"] Oct 30 06:54:24.930449 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.930420 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr"] Oct 30 06:54:24.933767 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:24.933743 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5mhqr"] Oct 30 06:54:25.338970 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:25.338938 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" path="/var/lib/kubelet/pods/55c1bc35-ccc6-43d1-9694-494bba09ab8e/volumes" Oct 30 06:54:25.339494 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:25.339477 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" path="/var/lib/kubelet/pods/8edcace5-3093-479b-9b0f-3d89251b214a/volumes" Oct 30 06:54:25.339929 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:25.339915 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" path="/var/lib/kubelet/pods/bd2b3564-82c5-4f7a-8b4e-9bdec23c5632/volumes" Oct 30 06:54:25.340681 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:25.340665 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" path="/var/lib/kubelet/pods/f98ee5c8-0953-4edf-8207-3db4f00c709b/volumes" Oct 30 06:54:33.701437 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:54:33.701403 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:54:44.089738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.089704 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch"] Oct 30 06:54:44.090330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090295 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" Oct 30 06:54:44.090330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090324 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" Oct 30 06:54:44.090330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090336 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090342 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090353 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090358 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090366 4431 cpu_manager.go:414] "containerMap: removing container" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090370 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090381 4431 cpu_manager.go:414] "containerMap: removing container" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090386 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090403 4431 cpu_manager.go:414] "containerMap: removing container" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090408 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090423 4431 cpu_manager.go:414] "containerMap: removing container" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090428 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090518 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2b3564-82c5-4f7a-8b4e-9bdec23c5632" containerName="istio-proxy" Oct 30 06:54:44.090526 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090530 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58609e0-908f-4c1a-93f8-a4cde0954868" containerName="istio-proxy" Oct 30 06:54:44.090960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090536 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edcace5-3093-479b-9b0f-3d89251b214a" containerName="istio-proxy" Oct 30 06:54:44.090960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090543 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98ee5c8-0953-4edf-8207-3db4f00c709b" containerName="istio-proxy" Oct 30 06:54:44.090960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090548 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c1bc35-ccc6-43d1-9694-494bba09ab8e" containerName="istio-proxy" Oct 30 06:54:44.090960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090554 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" Oct 30 06:54:44.090960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.090561 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9773d54c-d0fa-441b-b9b7-fe731d630d3a" containerName="istio-proxy" Oct 30 06:54:44.095958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.095940 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.113879 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.113854 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch"] Oct 30 06:54:44.122682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.122647 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b22a13-openshift-default-dockercfg-scj6c\"" Oct 30 06:54:44.201049 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.201018 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.201197 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.201090 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ws6\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-kube-api-access-68ws6\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.201197 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.201153 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c977e328-7791-4981-82c6-0e48d9e19b83-istio-podinfo\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.201339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.201233 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-certs\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.201408 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.201366 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-credential-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.201471 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.201402 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-data\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.201531 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.201506 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-envoy\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.201621 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.201607 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c977e328-7791-4981-82c6-0e48d9e19b83-istiod-ca-cert\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.201660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.201641 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-istio-token\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303138 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303115 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303281 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303188 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ws6\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-kube-api-access-68ws6\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303385 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303291 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c977e328-7791-4981-82c6-0e48d9e19b83-istio-podinfo\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303451 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303411 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-certs\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303505 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303465 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-credential-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303505 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303492 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-data\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303545 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-envoy\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303620 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303612 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303642 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c977e328-7791-4981-82c6-0e48d9e19b83-istiod-ca-cert\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303729 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303679 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-istio-token\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303835 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303786 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-certs\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303894 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303849 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-credential-socket\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.303932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.303901 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-data\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.304288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.304264 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c977e328-7791-4981-82c6-0e48d9e19b83-istiod-ca-cert\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.305834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.305810 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-envoy\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.305953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.305933 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c977e328-7791-4981-82c6-0e48d9e19b83-istio-podinfo\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.326230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.326202 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ws6\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-kube-api-access-68ws6\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.326370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.326352 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-istio-token\") pod \"e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.412948 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.412578 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:44.764801 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.764774 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch"] Oct 30 06:54:44.765664 ip-10-0-136-123 kubenswrapper[4431]: W1030 06:54:44.765639 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc977e328_7791_4981_82c6_0e48d9e19b83.slice/crio-cd9eebceb5ab85a2b81d593aca2890df5cf5a506df9ec1e12fffcdfe1cb53b0e WatchSource:0}: Error finding container cd9eebceb5ab85a2b81d593aca2890df5cf5a506df9ec1e12fffcdfe1cb53b0e: Status 404 returned error can't find the container with id cd9eebceb5ab85a2b81d593aca2890df5cf5a506df9ec1e12fffcdfe1cb53b0e Oct 30 06:54:44.768327 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.768276 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:54:44.768400 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.768379 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:54:44.768435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.768422 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 06:54:44.862932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.862893 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" event={"ID":"c977e328-7791-4981-82c6-0e48d9e19b83","Type":"ContainerStarted","Data":"7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c"} Oct 30 06:54:44.863110 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.862956 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" event={"ID":"c977e328-7791-4981-82c6-0e48d9e19b83","Type":"ContainerStarted","Data":"cd9eebceb5ab85a2b81d593aca2890df5cf5a506df9ec1e12fffcdfe1cb53b0e"} Oct 30 06:54:44.911417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:44.911354 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" podStartSLOduration=0.911334776 podStartE2EDuration="911.334776ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:54:44.908595843 +0000 UTC m=+3693.715117592" watchObservedRunningTime="2026-10-30 06:54:44.911334776 +0000 UTC m=+3693.717856524" Oct 30 06:54:45.419135 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:45.419093 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:45.421104 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:45.421074 4431 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:54:45.421235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:45.421133 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 06:54:46.419502 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:46.419469 4431 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:54:46.419930 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:46.419518 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 06:54:47.419508 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:47.419473 4431 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:54:47.419902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:47.419533 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 06:54:48.419070 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:48.419036 4431 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:54:48.419255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:48.419091 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 06:54:49.342847 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.342776 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w"] Oct 30 06:54:49.343632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.343001 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" podUID="9d96bcc0-783d-466d-9f38-820842153c58" containerName="maas-api" containerID="cri-o://799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04" gracePeriod=30 Oct 30 06:54:49.418777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.418743 4431 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:54:49.418937 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.418800 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 06:54:49.883328 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.871595 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:54:49.930876 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.930830 4431 generic.go:356] "Generic (PLEG): container finished" podID="9d96bcc0-783d-466d-9f38-820842153c58" containerID="799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04" exitCode=0 Oct 30 06:54:49.931098 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.931075 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" event={"ID":"9d96bcc0-783d-466d-9f38-820842153c58","Type":"ContainerDied","Data":"799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04"} Oct 30 06:54:49.931173 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.931140 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" event={"ID":"9d96bcc0-783d-466d-9f38-820842153c58","Type":"ContainerDied","Data":"d4bc1c0e001947fec1ed0ffdb46689ce1814b994c6edee90330d5253aae7cf9c"} Oct 30 06:54:49.931237 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.931175 4431 scope.go:122] "RemoveContainer" containerID="799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04" Oct 30 06:54:49.934017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.932280 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d96bcc0-783d-466d-9f38-820842153c58-kube-api-access-j69bz\" (UniqueName: \"kubernetes.io/projected/9d96bcc0-783d-466d-9f38-820842153c58-kube-api-access-j69bz\") pod \"9d96bcc0-783d-466d-9f38-820842153c58\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " Oct 30 06:54:49.934017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.932435 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-metrics-certs\") pod \"9d96bcc0-783d-466d-9f38-820842153c58\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " Oct 30 06:54:49.934017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.932554 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d96bcc0-783d-466d-9f38-820842153c58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d96bcc0-783d-466d-9f38-820842153c58-openshift-service-ca\") pod \"9d96bcc0-783d-466d-9f38-820842153c58\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " Oct 30 06:54:49.934017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.932658 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-maas-api-tls\") pod \"9d96bcc0-783d-466d-9f38-820842153c58\" (UID: \"9d96bcc0-783d-466d-9f38-820842153c58\") " Oct 30 06:54:49.934017 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.933969 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d96bcc0-783d-466d-9f38-820842153c58-openshift-service-ca" pod "9d96bcc0-783d-466d-9f38-820842153c58" (UID: "9d96bcc0-783d-466d-9f38-820842153c58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:49.940063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.937468 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w" Oct 30 06:54:49.940063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.939647 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-metrics-certs" pod "9d96bcc0-783d-466d-9f38-820842153c58" (UID: "9d96bcc0-783d-466d-9f38-820842153c58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:54:49.940063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.940052 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-maas-api-tls" pod "9d96bcc0-783d-466d-9f38-820842153c58" (UID: "9d96bcc0-783d-466d-9f38-820842153c58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:54:49.940479 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:49.940436 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d96bcc0-783d-466d-9f38-820842153c58-kube-api-access-j69bz" pod "9d96bcc0-783d-466d-9f38-820842153c58" (UID: "9d96bcc0-783d-466d-9f38-820842153c58"). InnerVolumeSpecName "kube-api-access-j69bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:50.001289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.001261 4431 scope.go:122] "RemoveContainer" containerID="799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04" Oct 30 06:54:50.001790 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:54:50.001759 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04\": container with ID starting with 799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04 not found: ID does not exist" containerID="799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04" Oct 30 06:54:50.001880 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.001803 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04"} err="failed to get container status \"799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04\": rpc error: code = NotFound desc = could not find container \"799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04\": container with ID starting with 799a7014e487aec18a0a5f02248e26121603c01ac80827342e29cf834f33fe04 not found: ID does not exist" Oct 30 06:54:50.035338 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.035287 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69bz\" (UniqueName: \"kubernetes.io/projected/9d96bcc0-783d-466d-9f38-820842153c58-kube-api-access-j69bz\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:50.035504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.035346 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:50.035504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.035366 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d96bcc0-783d-466d-9f38-820842153c58-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:50.035504 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.035383 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d96bcc0-783d-466d-9f38-820842153c58-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:54:50.301896 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.301816 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w"] Oct 30 06:54:50.315661 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.315629 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-eaee88-769bcbfc96-8qb6w"] Oct 30 06:54:50.419757 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.419717 4431 patch_prober.go:29] interesting pod/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:54:50.420161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:50.419789 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 06:54:51.339456 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:51.339423 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d96bcc0-783d-466d-9f38-820842153c58" path="/var/lib/kubelet/pods/9d96bcc0-783d-466d-9f38-820842153c58/volumes" Oct 30 06:54:51.424571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:51.424545 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:51.424917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:51.424893 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:51.425586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:51.425569 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:54:57.684573 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:57.684540 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx"] Oct 30 06:54:57.685144 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:54:57.684760 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" containerID="cri-o://15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed" gracePeriod=30 Oct 30 06:55:02.956736 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:02.956677 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:55:03.024088 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.024056 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-data\") pod \"48caab37-f6bb-49eb-8ac9-1390b217079f\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " Oct 30 06:55:03.024265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.024187 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-kube-api-access-r9g9m\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-kube-api-access-r9g9m\") pod \"48caab37-f6bb-49eb-8ac9-1390b217079f\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " Oct 30 06:55:03.024265 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.024245 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-certs\") pod \"48caab37-f6bb-49eb-8ac9-1390b217079f\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " Oct 30 06:55:03.024427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.024281 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-credential-socket\") pod \"48caab37-f6bb-49eb-8ac9-1390b217079f\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " Oct 30 06:55:03.024427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.024371 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-podinfo\") pod \"48caab37-f6bb-49eb-8ac9-1390b217079f\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " Oct 30 06:55:03.024551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.024469 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-envoy\") pod \"48caab37-f6bb-49eb-8ac9-1390b217079f\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " Oct 30 06:55:03.024551 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.024526 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-socket\") pod \"48caab37-f6bb-49eb-8ac9-1390b217079f\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " Oct 30 06:55:03.024640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.024587 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-token\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-token\") pod \"48caab37-f6bb-49eb-8ac9-1390b217079f\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " Oct 30 06:55:03.024640 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.024614 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48caab37-f6bb-49eb-8ac9-1390b217079f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48caab37-f6bb-49eb-8ac9-1390b217079f-istiod-ca-cert\") pod \"48caab37-f6bb-49eb-8ac9-1390b217079f\" (UID: \"48caab37-f6bb-49eb-8ac9-1390b217079f\") " Oct 30 06:55:03.025494 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.025464 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48caab37-f6bb-49eb-8ac9-1390b217079f-istiod-ca-cert" pod "48caab37-f6bb-49eb-8ac9-1390b217079f" (UID: "48caab37-f6bb-49eb-8ac9-1390b217079f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:55:03.025745 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.025727 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-data" pod "48caab37-f6bb-49eb-8ac9-1390b217079f" (UID: "48caab37-f6bb-49eb-8ac9-1390b217079f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.026612 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.026585 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-certs" pod "48caab37-f6bb-49eb-8ac9-1390b217079f" (UID: "48caab37-f6bb-49eb-8ac9-1390b217079f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.026960 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.026924 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-credential-socket" pod "48caab37-f6bb-49eb-8ac9-1390b217079f" (UID: "48caab37-f6bb-49eb-8ac9-1390b217079f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.030010 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.029972 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-kube-api-access-r9g9m" pod "48caab37-f6bb-49eb-8ac9-1390b217079f" (UID: "48caab37-f6bb-49eb-8ac9-1390b217079f"). InnerVolumeSpecName "kube-api-access-r9g9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:03.032340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.032313 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-socket" pod "48caab37-f6bb-49eb-8ac9-1390b217079f" (UID: "48caab37-f6bb-49eb-8ac9-1390b217079f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.033372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.033343 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-podinfo" pod "48caab37-f6bb-49eb-8ac9-1390b217079f" (UID: "48caab37-f6bb-49eb-8ac9-1390b217079f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:55:03.034802 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.034773 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-token" pod "48caab37-f6bb-49eb-8ac9-1390b217079f" (UID: "48caab37-f6bb-49eb-8ac9-1390b217079f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:03.035561 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.035533 4431 generic.go:356] "Generic (PLEG): container finished" podID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerID="15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed" exitCode=0 Oct 30 06:55:03.035714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.035696 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" event={"ID":"48caab37-f6bb-49eb-8ac9-1390b217079f","Type":"ContainerDied","Data":"15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed"} Oct 30 06:55:03.035794 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.035781 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" event={"ID":"48caab37-f6bb-49eb-8ac9-1390b217079f","Type":"ContainerDied","Data":"d5b3fce7f1f7282dafdc8d07d41b0bf1aeaeccf05ed5cf7cd490c1f24ea210c4"} Oct 30 06:55:03.035852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.035810 4431 scope.go:122] "RemoveContainer" containerID="15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed" Oct 30 06:55:03.035966 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.035954 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx" Oct 30 06:55:03.043026 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.043002 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-envoy" pod "48caab37-f6bb-49eb-8ac9-1390b217079f" (UID: "48caab37-f6bb-49eb-8ac9-1390b217079f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.064241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.064221 4431 scope.go:122] "RemoveContainer" containerID="15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed" Oct 30 06:55:03.064623 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:55:03.064600 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed\": container with ID starting with 15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed not found: ID does not exist" containerID="15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed" Oct 30 06:55:03.064707 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.064629 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed"} err="failed to get container status \"15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed\": rpc error: code = NotFound desc = could not find container \"15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed\": container with ID starting with 15835c907c25b7a1154a06772989e6fa9243e92fae42c06d7599774fede3feed not found: ID does not exist" Oct 30 06:55:03.126825 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.126786 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.126825 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.126819 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.127043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.126833 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48caab37-f6bb-49eb-8ac9-1390b217079f-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.127043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.126847 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.127043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.126859 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9g9m\" (UniqueName: \"kubernetes.io/projected/48caab37-f6bb-49eb-8ac9-1390b217079f-kube-api-access-r9g9m\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.127043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.126871 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.127043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.126883 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.127043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.126896 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.127043 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.126908 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48caab37-f6bb-49eb-8ac9-1390b217079f-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.384178 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.384147 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx"] Oct 30 06:55:03.388427 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:03.388397 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-5zmvx"] Oct 30 06:55:03.701991 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:55:03.701959 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:55:05.345611 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:05.345578 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" path="/var/lib/kubelet/pods/48caab37-f6bb-49eb-8ac9-1390b217079f/volumes" Oct 30 06:55:15.431118 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:15.431086 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r"] Oct 30 06:55:20.432003 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:20.431957 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" podUID="8a59b404-db18-4117-a038-181392be01d6" containerName="payload-processing" containerID="cri-o://1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3" gracePeriod=25 Oct 30 06:55:23.533841 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:23.531372 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" podUID="8a59b404-db18-4117-a038-181392be01d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:55:33.529811 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:33.529774 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" podUID="8a59b404-db18-4117-a038-181392be01d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:55:33.700685 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:55:33.700646 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:55:40.713274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:40.713245 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:55:40.807357 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:40.807294 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a59b404-db18-4117-a038-181392be01d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a59b404-db18-4117-a038-181392be01d6-plugins-config-volume\") pod \"8a59b404-db18-4117-a038-181392be01d6\" (UID: \"8a59b404-db18-4117-a038-181392be01d6\") " Oct 30 06:55:40.807549 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:40.807410 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a59b404-db18-4117-a038-181392be01d6-kube-api-access-sw5b9\" (UniqueName: \"kubernetes.io/projected/8a59b404-db18-4117-a038-181392be01d6-kube-api-access-sw5b9\") pod \"8a59b404-db18-4117-a038-181392be01d6\" (UID: \"8a59b404-db18-4117-a038-181392be01d6\") " Oct 30 06:55:40.807685 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:40.807657 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a59b404-db18-4117-a038-181392be01d6-plugins-config-volume" pod "8a59b404-db18-4117-a038-181392be01d6" (UID: "8a59b404-db18-4117-a038-181392be01d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:55:40.808067 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:40.808046 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a59b404-db18-4117-a038-181392be01d6-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:40.809630 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:40.809594 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a59b404-db18-4117-a038-181392be01d6-kube-api-access-sw5b9" pod "8a59b404-db18-4117-a038-181392be01d6" (UID: "8a59b404-db18-4117-a038-181392be01d6"). InnerVolumeSpecName "kube-api-access-sw5b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:40.909355 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:40.909320 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5b9\" (UniqueName: \"kubernetes.io/projected/8a59b404-db18-4117-a038-181392be01d6-kube-api-access-sw5b9\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:55:41.306235 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:41.306201 4431 generic.go:356] "Generic (PLEG): container finished" podID="8a59b404-db18-4117-a038-181392be01d6" containerID="1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3" exitCode=0 Oct 30 06:55:41.306452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:41.306253 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" Oct 30 06:55:41.306452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:41.306284 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" event={"ID":"8a59b404-db18-4117-a038-181392be01d6","Type":"ContainerDied","Data":"1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3"} Oct 30 06:55:41.306452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:41.306361 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r" event={"ID":"8a59b404-db18-4117-a038-181392be01d6","Type":"ContainerDied","Data":"87b8782731dc3fb0bd281b633804ab4b2a60490c2d0cdde1608168656035f56e"} Oct 30 06:55:41.306452 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:41.306383 4431 scope.go:122] "RemoveContainer" containerID="1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3" Oct 30 06:55:41.317511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:41.317431 4431 scope.go:122] "RemoveContainer" containerID="1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3" Oct 30 06:55:41.317791 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:55:41.317761 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3\": container with ID starting with 1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3 not found: ID does not exist" containerID="1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3" Oct 30 06:55:41.317969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:41.317802 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3"} err="failed to get container status \"1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3\": rpc error: code = NotFound desc = could not find container \"1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3\": container with ID starting with 1efbe5cbd9a235f6df99212099545fc019cb02d947056513d30a1ffd4572c2d3 not found: ID does not exist" Oct 30 06:55:41.358084 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:41.358056 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r"] Oct 30 06:55:41.370142 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:41.370117 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-88acce-545647888d-gd22r"] Oct 30 06:55:43.338033 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:55:43.338003 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a59b404-db18-4117-a038-181392be01d6" path="/var/lib/kubelet/pods/8a59b404-db18-4117-a038-181392be01d6/volumes" Oct 30 06:56:01.503042 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:01.503011 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s"] Oct 30 06:56:01.534035 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:01.534003 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb"] Oct 30 06:56:03.701528 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:56:03.701491 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:56:06.503703 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:06.503653 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" podUID="9c4debb0-8735-48b2-a4fd-2bb725083af0" containerName="payload-processing" containerID="cri-o://dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b" gracePeriod=25 Oct 30 06:56:06.534958 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:06.534918 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" podUID="2d53b413-657e-4353-b14e-75ccd742e78b" containerName="payload-pre-processing" containerID="cri-o://b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e" gracePeriod=25 Oct 30 06:56:15.867173 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:15.867131 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" podUID="2d53b413-657e-4353-b14e-75ccd742e78b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:56:15.867661 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:15.867176 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" podUID="9c4debb0-8735-48b2-a4fd-2bb725083af0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:56:25.867208 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:25.867123 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" podUID="2d53b413-657e-4353-b14e-75ccd742e78b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:56:25.867704 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:25.867124 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" podUID="9c4debb0-8735-48b2-a4fd-2bb725083af0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:56:26.806728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.806695 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:56:26.840834 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.840808 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:56:26.952148 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.952114 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4debb0-8735-48b2-a4fd-2bb725083af0-kube-api-access-6d99c\" (UniqueName: \"kubernetes.io/projected/9c4debb0-8735-48b2-a4fd-2bb725083af0-kube-api-access-6d99c\") pod \"9c4debb0-8735-48b2-a4fd-2bb725083af0\" (UID: \"9c4debb0-8735-48b2-a4fd-2bb725083af0\") " Oct 30 06:56:26.952682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.952161 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4debb0-8735-48b2-a4fd-2bb725083af0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c4debb0-8735-48b2-a4fd-2bb725083af0-plugins-config-volume\") pod \"9c4debb0-8735-48b2-a4fd-2bb725083af0\" (UID: \"9c4debb0-8735-48b2-a4fd-2bb725083af0\") " Oct 30 06:56:26.952682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.952225 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d53b413-657e-4353-b14e-75ccd742e78b-kube-api-access-c2trf\" (UniqueName: \"kubernetes.io/projected/2d53b413-657e-4353-b14e-75ccd742e78b-kube-api-access-c2trf\") pod \"2d53b413-657e-4353-b14e-75ccd742e78b\" (UID: \"2d53b413-657e-4353-b14e-75ccd742e78b\") " Oct 30 06:56:26.952682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.952359 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d53b413-657e-4353-b14e-75ccd742e78b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d53b413-657e-4353-b14e-75ccd742e78b-plugins-config-volume\") pod \"2d53b413-657e-4353-b14e-75ccd742e78b\" (UID: \"2d53b413-657e-4353-b14e-75ccd742e78b\") " Oct 30 06:56:26.952682 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.952662 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4debb0-8735-48b2-a4fd-2bb725083af0-plugins-config-volume" pod "9c4debb0-8735-48b2-a4fd-2bb725083af0" (UID: "9c4debb0-8735-48b2-a4fd-2bb725083af0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:26.952907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.952720 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d53b413-657e-4353-b14e-75ccd742e78b-plugins-config-volume" pod "2d53b413-657e-4353-b14e-75ccd742e78b" (UID: "2d53b413-657e-4353-b14e-75ccd742e78b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:26.952907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.952875 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d53b413-657e-4353-b14e-75ccd742e78b-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:26.952907 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.952895 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c4debb0-8735-48b2-a4fd-2bb725083af0-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:26.954523 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.954498 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d53b413-657e-4353-b14e-75ccd742e78b-kube-api-access-c2trf" pod "2d53b413-657e-4353-b14e-75ccd742e78b" (UID: "2d53b413-657e-4353-b14e-75ccd742e78b"). InnerVolumeSpecName "kube-api-access-c2trf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:26.954840 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:26.954822 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4debb0-8735-48b2-a4fd-2bb725083af0-kube-api-access-6d99c" pod "9c4debb0-8735-48b2-a4fd-2bb725083af0" (UID: "9c4debb0-8735-48b2-a4fd-2bb725083af0"). InnerVolumeSpecName "kube-api-access-6d99c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:27.054104 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.054027 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d99c\" (UniqueName: \"kubernetes.io/projected/9c4debb0-8735-48b2-a4fd-2bb725083af0-kube-api-access-6d99c\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:27.054104 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.054054 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2trf\" (UniqueName: \"kubernetes.io/projected/2d53b413-657e-4353-b14e-75ccd742e78b-kube-api-access-c2trf\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:27.588969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.588933 4431 generic.go:356] "Generic (PLEG): container finished" podID="2d53b413-657e-4353-b14e-75ccd742e78b" containerID="b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e" exitCode=0 Oct 30 06:56:27.589131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.588985 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" Oct 30 06:56:27.589131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.589022 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" event={"ID":"2d53b413-657e-4353-b14e-75ccd742e78b","Type":"ContainerDied","Data":"b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e"} Oct 30 06:56:27.589131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.589089 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb" event={"ID":"2d53b413-657e-4353-b14e-75ccd742e78b","Type":"ContainerDied","Data":"0fbfefa8fceaac166f72015228257046309fa99abdd6b836b5d94c5dd1267d06"} Oct 30 06:56:27.589131 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.589118 4431 scope.go:122] "RemoveContainer" containerID="b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e" Oct 30 06:56:27.590582 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.590560 4431 generic.go:356] "Generic (PLEG): container finished" podID="9c4debb0-8735-48b2-a4fd-2bb725083af0" containerID="dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b" exitCode=0 Oct 30 06:56:27.590712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.590602 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" Oct 30 06:56:27.590712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.590647 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" event={"ID":"9c4debb0-8735-48b2-a4fd-2bb725083af0","Type":"ContainerDied","Data":"dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b"} Oct 30 06:56:27.590712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.590704 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s" event={"ID":"9c4debb0-8735-48b2-a4fd-2bb725083af0","Type":"ContainerDied","Data":"9f5663f7a97e2208c512264795fd6baa5dcbbd7a60b4eb5d4c58442469694bfd"} Oct 30 06:56:27.599197 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.599183 4431 scope.go:122] "RemoveContainer" containerID="b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e" Oct 30 06:56:27.599595 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:56:27.599564 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e\": container with ID starting with b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e not found: ID does not exist" containerID="b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e" Oct 30 06:56:27.599677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.599601 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e"} err="failed to get container status \"b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e\": rpc error: code = NotFound desc = could not find container \"b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e\": container with ID starting with b3c89efb9b441af39a871112da79623d1d1bd8d1231b18e79c04bedc49d3a90e not found: ID does not exist" Oct 30 06:56:27.599677 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.599619 4431 scope.go:122] "RemoveContainer" containerID="dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b" Oct 30 06:56:27.608214 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.608198 4431 scope.go:122] "RemoveContainer" containerID="dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b" Oct 30 06:56:27.608473 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:56:27.608456 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b\": container with ID starting with dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b not found: ID does not exist" containerID="dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b" Oct 30 06:56:27.608517 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.608480 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b"} err="failed to get container status \"dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b\": rpc error: code = NotFound desc = could not find container \"dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b\": container with ID starting with dc32f86eeb13f57fba8de2a4751b9191364c673485d0e674606a19a03bd3547b not found: ID does not exist" Oct 30 06:56:27.643133 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.643109 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s"] Oct 30 06:56:27.646740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.646716 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c44730-7cb74987b-2q97s"] Oct 30 06:56:27.679809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.679778 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb"] Oct 30 06:56:27.683431 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:27.683410 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c44730-57d47f9459-m68pb"] Oct 30 06:56:29.337998 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:29.337959 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d53b413-657e-4353-b14e-75ccd742e78b" path="/var/lib/kubelet/pods/2d53b413-657e-4353-b14e-75ccd742e78b/volumes" Oct 30 06:56:29.338523 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:29.338411 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4debb0-8735-48b2-a4fd-2bb725083af0" path="/var/lib/kubelet/pods/9c4debb0-8735-48b2-a4fd-2bb725083af0/volumes" Oct 30 06:56:31.960094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:31.960056 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt"] Oct 30 06:56:33.701417 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:56:33.701376 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:56:46.981053 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:46.981015 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" podUID="6829b5b1-5091-4483-be7f-dd73b6a4428e" containerName="main" containerID="cri-o://b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29" gracePeriod=45 Oct 30 06:56:47.248653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.248598 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:56:47.318295 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.318257 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-dshm\") pod \"6829b5b1-5091-4483-be7f-dd73b6a4428e\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " Oct 30 06:56:47.318474 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.318377 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6829b5b1-5091-4483-be7f-dd73b6a4428e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6829b5b1-5091-4483-be7f-dd73b6a4428e-tls-certs\") pod \"6829b5b1-5091-4483-be7f-dd73b6a4428e\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " Oct 30 06:56:47.318558 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.318540 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6829b5b1-5091-4483-be7f-dd73b6a4428e-kube-api-access-mrhxk\" (UniqueName: \"kubernetes.io/projected/6829b5b1-5091-4483-be7f-dd73b6a4428e-kube-api-access-mrhxk\") pod \"6829b5b1-5091-4483-be7f-dd73b6a4428e\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " Oct 30 06:56:47.318616 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.318582 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-home\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-home\") pod \"6829b5b1-5091-4483-be7f-dd73b6a4428e\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " Oct 30 06:56:47.318712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.318694 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-model-cache\") pod \"6829b5b1-5091-4483-be7f-dd73b6a4428e\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " Oct 30 06:56:47.318786 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.318734 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-tmp-dir\") pod \"6829b5b1-5091-4483-be7f-dd73b6a4428e\" (UID: \"6829b5b1-5091-4483-be7f-dd73b6a4428e\") " Oct 30 06:56:47.319011 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.318974 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-home" pod "6829b5b1-5091-4483-be7f-dd73b6a4428e" (UID: "6829b5b1-5091-4483-be7f-dd73b6a4428e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:56:47.319011 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.318981 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-model-cache" pod "6829b5b1-5091-4483-be7f-dd73b6a4428e" (UID: "6829b5b1-5091-4483-be7f-dd73b6a4428e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:56:47.319230 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.319205 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-tmp-dir" pod "6829b5b1-5091-4483-be7f-dd73b6a4428e" (UID: "6829b5b1-5091-4483-be7f-dd73b6a4428e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:56:47.319421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.319395 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:47.319530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.319425 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:47.321222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.321194 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6829b5b1-5091-4483-be7f-dd73b6a4428e-kube-api-access-mrhxk" pod "6829b5b1-5091-4483-be7f-dd73b6a4428e" (UID: "6829b5b1-5091-4483-be7f-dd73b6a4428e"). InnerVolumeSpecName "kube-api-access-mrhxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:47.321349 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.321200 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6829b5b1-5091-4483-be7f-dd73b6a4428e-tls-certs" pod "6829b5b1-5091-4483-be7f-dd73b6a4428e" (UID: "6829b5b1-5091-4483-be7f-dd73b6a4428e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:47.321484 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.321461 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-dshm" pod "6829b5b1-5091-4483-be7f-dd73b6a4428e" (UID: "6829b5b1-5091-4483-be7f-dd73b6a4428e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:56:47.420571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.420538 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:47.420571 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.420569 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6829b5b1-5091-4483-be7f-dd73b6a4428e-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:47.420782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.420579 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6829b5b1-5091-4483-be7f-dd73b6a4428e-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:47.420782 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.420587 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrhxk\" (UniqueName: \"kubernetes.io/projected/6829b5b1-5091-4483-be7f-dd73b6a4428e-kube-api-access-mrhxk\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:56:47.751871 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.751835 4431 generic.go:356] "Generic (PLEG): container finished" podID="6829b5b1-5091-4483-be7f-dd73b6a4428e" containerID="b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29" exitCode=0 Oct 30 06:56:47.752038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.751882 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" Oct 30 06:56:47.752038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.751911 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" event={"ID":"6829b5b1-5091-4483-be7f-dd73b6a4428e","Type":"ContainerDied","Data":"b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29"} Oct 30 06:56:47.752038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.751949 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" event={"ID":"6829b5b1-5091-4483-be7f-dd73b6a4428e","Type":"ContainerDied","Data":"39db005f22655a6aa8b3db14c5310542d872d14bae72bf25c39c891a93b3eb17"} Oct 30 06:56:47.752038 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.751968 4431 scope.go:122] "RemoveContainer" containerID="b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29" Oct 30 06:56:47.761519 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.761502 4431 scope.go:122] "RemoveContainer" containerID="b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29" Oct 30 06:56:47.761787 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:56:47.761767 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29\": container with ID starting with b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29 not found: ID does not exist" containerID="b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29" Oct 30 06:56:47.761852 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.761800 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29"} err="failed to get container status \"b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29\": rpc error: code = NotFound desc = could not find container \"b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29\": container with ID starting with b4a4b8ff1350aeed07b5e33c27baaa4f4131c053fdd1cb84a325467133e76c29 not found: ID does not exist" Oct 30 06:56:47.803165 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.803142 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt"] Oct 30 06:56:47.807379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:47.807359 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt"] Oct 30 06:56:48.234050 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:48.234008 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b22a13/test-model-b22a13-kserve-d7896b77-lk7mt" podUID="6829b5b1-5091-4483-be7f-dd73b6a4428e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.175:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:56:49.338162 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:56:49.338131 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6829b5b1-5091-4483-be7f-dd73b6a4428e" path="/var/lib/kubelet/pods/6829b5b1-5091-4483-be7f-dd73b6a4428e/volumes" Oct 30 06:57:03.701607 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:03.701534 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:57:13.461480 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.461432 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl"] Oct 30 06:57:13.462006 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.461713 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" podUID="df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" containerName="maas-api" containerID="cri-o://07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356" gracePeriod=30 Oct 30 06:57:13.583137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.583103 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" podUID="df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": dial tcp 10.133.0.168:8443: connect: connection refused" Oct 30 06:57:13.794349 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.793451 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:57:13.903859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.903822 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-maas-api-tls\") pod \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " Oct 30 06:57:13.904056 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.903990 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-metrics-certs\") pod \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " Oct 30 06:57:13.904056 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.904035 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-openshift-service-ca\") pod \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " Oct 30 06:57:13.904176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.904151 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-kube-api-access-w4lpz\" (UniqueName: \"kubernetes.io/projected/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-kube-api-access-w4lpz\") pod \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\" (UID: \"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb\") " Oct 30 06:57:13.904548 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.904506 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-openshift-service-ca" pod "df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" (UID: "df9fb6fa-fbde-4cc5-b43d-8856815fb3cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:13.904760 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.904736 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:13.906055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.906023 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-maas-api-tls" pod "df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" (UID: "df9fb6fa-fbde-4cc5-b43d-8856815fb3cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:13.906185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.906126 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-metrics-certs" pod "df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" (UID: "df9fb6fa-fbde-4cc5-b43d-8856815fb3cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:13.906238 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:13.906219 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-kube-api-access-w4lpz" pod "df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" (UID: "df9fb6fa-fbde-4cc5-b43d-8856815fb3cb"). InnerVolumeSpecName "kube-api-access-w4lpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:14.010374 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.009787 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4lpz\" (UniqueName: \"kubernetes.io/projected/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-kube-api-access-w4lpz\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:14.039319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.035351 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:14.039319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.035390 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:14.052341 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.049446 4431 generic.go:356] "Generic (PLEG): container finished" podID="df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" containerID="07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356" exitCode=0 Oct 30 06:57:14.052341 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.049651 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" event={"ID":"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb","Type":"ContainerDied","Data":"07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356"} Oct 30 06:57:14.052341 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.049705 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" event={"ID":"df9fb6fa-fbde-4cc5-b43d-8856815fb3cb","Type":"ContainerDied","Data":"0e348a75471cc9b4940669db2e290384e9bba747d2153b64c75de8e99a6a69cf"} Oct 30 06:57:14.052341 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.049737 4431 scope.go:122] "RemoveContainer" containerID="07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356" Oct 30 06:57:14.052341 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.049904 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl" Oct 30 06:57:14.073999 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.073977 4431 scope.go:122] "RemoveContainer" containerID="07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356" Oct 30 06:57:14.074474 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:14.074447 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356\": container with ID starting with 07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356 not found: ID does not exist" containerID="07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356" Oct 30 06:57:14.074594 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.074484 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356"} err="failed to get container status \"07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356\": rpc error: code = NotFound desc = could not find container \"07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356\": container with ID starting with 07707163e7d899643d1ab0a2cbbc475a3fe09022446cdbfdec70e8eda1260356 not found: ID does not exist" Oct 30 06:57:14.112909 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.112877 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl"] Oct 30 06:57:14.118984 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:14.118960 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a13-557bb89f6-2jspl"] Oct 30 06:57:15.337543 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:15.337504 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" path="/var/lib/kubelet/pods/df9fb6fa-fbde-4cc5-b43d-8856815fb3cb/volumes" Oct 30 06:57:20.141993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:20.141954 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc"] Oct 30 06:57:20.142660 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:20.142220 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" containerID="cri-o://8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530" gracePeriod=30 Oct 30 06:57:23.402854 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:23.402815 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch"] Oct 30 06:57:23.403363 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:23.403094 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" containerID="cri-o://7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c" gracePeriod=30 Oct 30 06:57:25.424854 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.424829 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:57:25.538055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.537980 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-certs\") pod \"6d865d5a-0560-4d6e-ae24-6985104463f8\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " Oct 30 06:57:25.538055 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538029 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-kube-api-access-d6h5m\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-kube-api-access-d6h5m\") pod \"6d865d5a-0560-4d6e-ae24-6985104463f8\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " Oct 30 06:57:25.538244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538111 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-envoy\") pod \"6d865d5a-0560-4d6e-ae24-6985104463f8\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " Oct 30 06:57:25.538244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538145 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-podinfo\") pod \"6d865d5a-0560-4d6e-ae24-6985104463f8\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " Oct 30 06:57:25.538244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538176 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-credential-socket\") pod \"6d865d5a-0560-4d6e-ae24-6985104463f8\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " Oct 30 06:57:25.538244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538240 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-token\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-token\") pod \"6d865d5a-0560-4d6e-ae24-6985104463f8\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " Oct 30 06:57:25.538492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538248 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-certs" pod "6d865d5a-0560-4d6e-ae24-6985104463f8" (UID: "6d865d5a-0560-4d6e-ae24-6985104463f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:25.538492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538267 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d865d5a-0560-4d6e-ae24-6985104463f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d865d5a-0560-4d6e-ae24-6985104463f8-istiod-ca-cert\") pod \"6d865d5a-0560-4d6e-ae24-6985104463f8\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " Oct 30 06:57:25.538492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538291 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-socket\") pod \"6d865d5a-0560-4d6e-ae24-6985104463f8\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " Oct 30 06:57:25.538492 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538375 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-data\") pod \"6d865d5a-0560-4d6e-ae24-6985104463f8\" (UID: \"6d865d5a-0560-4d6e-ae24-6985104463f8\") " Oct 30 06:57:25.538768 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538527 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-credential-socket" pod "6d865d5a-0560-4d6e-ae24-6985104463f8" (UID: "6d865d5a-0560-4d6e-ae24-6985104463f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:25.538768 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538744 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-data" pod "6d865d5a-0560-4d6e-ae24-6985104463f8" (UID: "6d865d5a-0560-4d6e-ae24-6985104463f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:25.538768 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538757 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-socket" pod "6d865d5a-0560-4d6e-ae24-6985104463f8" (UID: "6d865d5a-0560-4d6e-ae24-6985104463f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:25.538962 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.538938 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d865d5a-0560-4d6e-ae24-6985104463f8-istiod-ca-cert" pod "6d865d5a-0560-4d6e-ae24-6985104463f8" (UID: "6d865d5a-0560-4d6e-ae24-6985104463f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:25.539045 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.539029 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.539108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.539047 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.539108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.539059 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.539108 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.539072 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.540505 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.540479 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-podinfo" pod "6d865d5a-0560-4d6e-ae24-6985104463f8" (UID: "6d865d5a-0560-4d6e-ae24-6985104463f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:57:25.540505 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.540493 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-token" pod "6d865d5a-0560-4d6e-ae24-6985104463f8" (UID: "6d865d5a-0560-4d6e-ae24-6985104463f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:25.540777 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.540756 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-envoy" pod "6d865d5a-0560-4d6e-ae24-6985104463f8" (UID: "6d865d5a-0560-4d6e-ae24-6985104463f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:25.540838 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.540758 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-kube-api-access-d6h5m" pod "6d865d5a-0560-4d6e-ae24-6985104463f8" (UID: "6d865d5a-0560-4d6e-ae24-6985104463f8"). InnerVolumeSpecName "kube-api-access-d6h5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:25.639917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.639886 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.639917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.639913 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d865d5a-0560-4d6e-ae24-6985104463f8-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.640059 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.639922 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6h5m\" (UniqueName: \"kubernetes.io/projected/6d865d5a-0560-4d6e-ae24-6985104463f8-kube-api-access-d6h5m\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.640059 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.639931 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:25.640059 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:25.639940 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d865d5a-0560-4d6e-ae24-6985104463f8-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:26.166783 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.166751 4431 generic.go:356] "Generic (PLEG): container finished" podID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerID="8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530" exitCode=0 Oct 30 06:57:26.166996 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.166814 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" Oct 30 06:57:26.166996 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.166829 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" event={"ID":"6d865d5a-0560-4d6e-ae24-6985104463f8","Type":"ContainerDied","Data":"8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530"} Oct 30 06:57:26.166996 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.166895 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc" event={"ID":"6d865d5a-0560-4d6e-ae24-6985104463f8","Type":"ContainerDied","Data":"eece24c410da84645e9210c31bb20df44aa7ff52769dfa7e822f3c338e198e1e"} Oct 30 06:57:26.166996 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.166928 4431 scope.go:122] "RemoveContainer" containerID="8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530" Oct 30 06:57:26.177986 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.177839 4431 scope.go:122] "RemoveContainer" containerID="8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be" Oct 30 06:57:26.187158 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.187139 4431 scope.go:122] "RemoveContainer" containerID="8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530" Oct 30 06:57:26.187527 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:26.187507 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530\": container with ID starting with 8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530 not found: ID does not exist" containerID="8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530" Oct 30 06:57:26.187591 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.187535 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530"} err="failed to get container status \"8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530\": rpc error: code = NotFound desc = could not find container \"8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530\": container with ID starting with 8ca50eff6df14071734f4c0720a79212c279f8c0af1f48b5716c99b2996c1530 not found: ID does not exist" Oct 30 06:57:26.187591 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.187553 4431 scope.go:122] "RemoveContainer" containerID="8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be" Oct 30 06:57:26.187806 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:26.187788 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be\": container with ID starting with 8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be not found: ID does not exist" containerID="8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be" Oct 30 06:57:26.187844 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.187813 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be"} err="failed to get container status \"8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be\": rpc error: code = NotFound desc = could not find container \"8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be\": container with ID starting with 8af00c7f62a1cf527570e1e22065e28940cf38b5ff038cf93a15d5eaaac725be not found: ID does not exist" Oct 30 06:57:26.230810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.230790 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc"] Oct 30 06:57:26.242931 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:26.242904 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4f3973-openshift-default-559bc8bf6c-zkbzc"] Oct 30 06:57:27.219330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:27.219272 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6"] Oct 30 06:57:27.219789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:27.219492 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" containerName="istio-proxy" containerID="cri-o://0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012" gracePeriod=30 Oct 30 06:57:27.341398 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:27.341367 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" path="/var/lib/kubelet/pods/6d865d5a-0560-4d6e-ae24-6985104463f8/volumes" Oct 30 06:57:28.661132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.661105 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:57:28.693132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693108 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-istio-token\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-istio-token\") pod \"c977e328-7791-4981-82c6-0e48d9e19b83\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " Oct 30 06:57:28.693244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693153 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-kube-api-access-68ws6\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-kube-api-access-68ws6\") pod \"c977e328-7791-4981-82c6-0e48d9e19b83\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " Oct 30 06:57:28.693244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693183 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-envoy\") pod \"c977e328-7791-4981-82c6-0e48d9e19b83\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " Oct 30 06:57:28.693244 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693215 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c977e328-7791-4981-82c6-0e48d9e19b83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c977e328-7791-4981-82c6-0e48d9e19b83-istio-podinfo\") pod \"c977e328-7791-4981-82c6-0e48d9e19b83\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " Oct 30 06:57:28.693437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693248 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c977e328-7791-4981-82c6-0e48d9e19b83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c977e328-7791-4981-82c6-0e48d9e19b83-istiod-ca-cert\") pod \"c977e328-7791-4981-82c6-0e48d9e19b83\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " Oct 30 06:57:28.693437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693284 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-credential-socket\") pod \"c977e328-7791-4981-82c6-0e48d9e19b83\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " Oct 30 06:57:28.693437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693351 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-certs\") pod \"c977e328-7791-4981-82c6-0e48d9e19b83\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " Oct 30 06:57:28.693437 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693404 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-socket\") pod \"c977e328-7791-4981-82c6-0e48d9e19b83\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " Oct 30 06:57:28.693656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693569 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-data\") pod \"c977e328-7791-4981-82c6-0e48d9e19b83\" (UID: \"c977e328-7791-4981-82c6-0e48d9e19b83\") " Oct 30 06:57:28.693656 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693631 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-certs" pod "c977e328-7791-4981-82c6-0e48d9e19b83" (UID: "c977e328-7791-4981-82c6-0e48d9e19b83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.693769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693713 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c977e328-7791-4981-82c6-0e48d9e19b83-istiod-ca-cert" pod "c977e328-7791-4981-82c6-0e48d9e19b83" (UID: "c977e328-7791-4981-82c6-0e48d9e19b83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:28.693855 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.693833 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-socket" pod "c977e328-7791-4981-82c6-0e48d9e19b83" (UID: "c977e328-7791-4981-82c6-0e48d9e19b83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.694072 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.694044 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-data" pod "c977e328-7791-4981-82c6-0e48d9e19b83" (UID: "c977e328-7791-4981-82c6-0e48d9e19b83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.694203 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.694118 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-credential-socket" pod "c977e328-7791-4981-82c6-0e48d9e19b83" (UID: "c977e328-7791-4981-82c6-0e48d9e19b83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.694274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.694202 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.694274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.694217 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c977e328-7791-4981-82c6-0e48d9e19b83-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.694274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.694227 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.694274 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.694235 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.695473 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.695446 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-istio-token" pod "c977e328-7791-4981-82c6-0e48d9e19b83" (UID: "c977e328-7791-4981-82c6-0e48d9e19b83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:28.695594 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.695554 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-kube-api-access-68ws6" pod "c977e328-7791-4981-82c6-0e48d9e19b83" (UID: "c977e328-7791-4981-82c6-0e48d9e19b83"). InnerVolumeSpecName "kube-api-access-68ws6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:28.695814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.695793 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c977e328-7791-4981-82c6-0e48d9e19b83-istio-podinfo" pod "c977e328-7791-4981-82c6-0e48d9e19b83" (UID: "c977e328-7791-4981-82c6-0e48d9e19b83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:57:28.695814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.695802 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-envoy" pod "c977e328-7791-4981-82c6-0e48d9e19b83" (UID: "c977e328-7791-4981-82c6-0e48d9e19b83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:28.794953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.794920 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.794953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.794946 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ws6\" (UniqueName: \"kubernetes.io/projected/c977e328-7791-4981-82c6-0e48d9e19b83-kube-api-access-68ws6\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.794953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.794956 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.794953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.794964 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c977e328-7791-4981-82c6-0e48d9e19b83-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:28.795317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:28.794973 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c977e328-7791-4981-82c6-0e48d9e19b83-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:29.187001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.186912 4431 generic.go:356] "Generic (PLEG): container finished" podID="c977e328-7791-4981-82c6-0e48d9e19b83" containerID="7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c" exitCode=0 Oct 30 06:57:29.187001 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.186963 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" Oct 30 06:57:29.187236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.187000 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" event={"ID":"c977e328-7791-4981-82c6-0e48d9e19b83","Type":"ContainerDied","Data":"7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c"} Oct 30 06:57:29.187236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.187060 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch" event={"ID":"c977e328-7791-4981-82c6-0e48d9e19b83","Type":"ContainerDied","Data":"cd9eebceb5ab85a2b81d593aca2890df5cf5a506df9ec1e12fffcdfe1cb53b0e"} Oct 30 06:57:29.187236 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.187084 4431 scope.go:122] "RemoveContainer" containerID="7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c" Oct 30 06:57:29.198027 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.198005 4431 scope.go:122] "RemoveContainer" containerID="7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c" Oct 30 06:57:29.198359 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:29.198329 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c\": container with ID starting with 7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c not found: ID does not exist" containerID="7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c" Oct 30 06:57:29.198461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.198370 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c"} err="failed to get container status \"7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c\": rpc error: code = NotFound desc = could not find container \"7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c\": container with ID starting with 7f89d2498c164356ebb0812e11ff90f532a221d484f286320884ee90a21c531c not found: ID does not exist" Oct 30 06:57:29.251027 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.250993 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch"] Oct 30 06:57:29.263142 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.263114 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b22a13-openshift-default-c6f496b49-kgnch"] Oct 30 06:57:29.338560 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.338529 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" path="/var/lib/kubelet/pods/c977e328-7791-4981-82c6-0e48d9e19b83/volumes" Oct 30 06:57:29.613962 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.613930 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt"] Oct 30 06:57:29.614185 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:29.614158 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" containerID="cri-o://e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7" gracePeriod=30 Oct 30 06:57:32.473581 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.473558 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:57:32.548917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.548827 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-certs\") pod \"8505f718-e49c-4231-91f0-20dd4a1be595\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " Oct 30 06:57:32.549085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.548945 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8505f718-e49c-4231-91f0-20dd4a1be595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8505f718-e49c-4231-91f0-20dd4a1be595-istiod-ca-cert\") pod \"8505f718-e49c-4231-91f0-20dd4a1be595\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " Oct 30 06:57:32.549085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.548994 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-envoy\") pod \"8505f718-e49c-4231-91f0-20dd4a1be595\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " Oct 30 06:57:32.549085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.549053 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-credential-socket\") pod \"8505f718-e49c-4231-91f0-20dd4a1be595\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " Oct 30 06:57:32.549257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.549086 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-istio-token\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-istio-token\") pod \"8505f718-e49c-4231-91f0-20dd4a1be595\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " Oct 30 06:57:32.549257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.549126 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-certs" pod "8505f718-e49c-4231-91f0-20dd4a1be595" (UID: "8505f718-e49c-4231-91f0-20dd4a1be595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:32.549257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.549138 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8505f718-e49c-4231-91f0-20dd4a1be595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8505f718-e49c-4231-91f0-20dd4a1be595-istio-podinfo\") pod \"8505f718-e49c-4231-91f0-20dd4a1be595\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " Oct 30 06:57:32.549463 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.549266 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-data\") pod \"8505f718-e49c-4231-91f0-20dd4a1be595\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " Oct 30 06:57:32.549463 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.549367 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-socket\") pod \"8505f718-e49c-4231-91f0-20dd4a1be595\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " Oct 30 06:57:32.549463 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.549396 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-kube-api-access-nqn8f\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-kube-api-access-nqn8f\") pod \"8505f718-e49c-4231-91f0-20dd4a1be595\" (UID: \"8505f718-e49c-4231-91f0-20dd4a1be595\") " Oct 30 06:57:32.550793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.549743 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-data" pod "8505f718-e49c-4231-91f0-20dd4a1be595" (UID: "8505f718-e49c-4231-91f0-20dd4a1be595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:32.550793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.550011 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:32.550793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.550034 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:32.550793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.550271 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-socket" pod "8505f718-e49c-4231-91f0-20dd4a1be595" (UID: "8505f718-e49c-4231-91f0-20dd4a1be595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:32.550793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.550535 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-credential-socket" pod "8505f718-e49c-4231-91f0-20dd4a1be595" (UID: "8505f718-e49c-4231-91f0-20dd4a1be595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:32.550793 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.550567 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8505f718-e49c-4231-91f0-20dd4a1be595-istiod-ca-cert" pod "8505f718-e49c-4231-91f0-20dd4a1be595" (UID: "8505f718-e49c-4231-91f0-20dd4a1be595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:32.553184 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.553144 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-envoy" pod "8505f718-e49c-4231-91f0-20dd4a1be595" (UID: "8505f718-e49c-4231-91f0-20dd4a1be595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:32.553417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.553381 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-istio-token" pod "8505f718-e49c-4231-91f0-20dd4a1be595" (UID: "8505f718-e49c-4231-91f0-20dd4a1be595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:32.553746 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.553722 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-kube-api-access-nqn8f" pod "8505f718-e49c-4231-91f0-20dd4a1be595" (UID: "8505f718-e49c-4231-91f0-20dd4a1be595"). InnerVolumeSpecName "kube-api-access-nqn8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:32.554463 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.554429 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8505f718-e49c-4231-91f0-20dd4a1be595-istio-podinfo" pod "8505f718-e49c-4231-91f0-20dd4a1be595" (UID: "8505f718-e49c-4231-91f0-20dd4a1be595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:57:32.651721 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.651689 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8505f718-e49c-4231-91f0-20dd4a1be595-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:32.651721 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.651718 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:32.651721 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.651727 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:32.651994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.651736 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:32.651994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.651744 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8505f718-e49c-4231-91f0-20dd4a1be595-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:32.651994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.651752 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8505f718-e49c-4231-91f0-20dd4a1be595-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:32.651994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:32.651765 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqn8f\" (UniqueName: \"kubernetes.io/projected/8505f718-e49c-4231-91f0-20dd4a1be595-kube-api-access-nqn8f\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:33.214031 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.213994 4431 generic.go:356] "Generic (PLEG): container finished" podID="8505f718-e49c-4231-91f0-20dd4a1be595" containerID="0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012" exitCode=0 Oct 30 06:57:33.214246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.214054 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" Oct 30 06:57:33.214246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.214075 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" event={"ID":"8505f718-e49c-4231-91f0-20dd4a1be595","Type":"ContainerDied","Data":"0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012"} Oct 30 06:57:33.214246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.214159 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6" event={"ID":"8505f718-e49c-4231-91f0-20dd4a1be595","Type":"ContainerDied","Data":"851495299d9ab66139bf4ebf8deb06226f1f2008a4c1ce200ca4043a177e4210"} Oct 30 06:57:33.214246 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.214184 4431 scope.go:122] "RemoveContainer" containerID="0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012" Oct 30 06:57:33.227675 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.227645 4431 scope.go:122] "RemoveContainer" containerID="0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012" Oct 30 06:57:33.227985 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:33.227958 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012\": container with ID starting with 0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012 not found: ID does not exist" containerID="0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012" Oct 30 06:57:33.228052 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.227995 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012"} err="failed to get container status \"0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012\": rpc error: code = NotFound desc = could not find container \"0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012\": container with ID starting with 0d28b06096c31e1d4df5fbc6ef17ed3a13693b7918b4f15efbcc9770fc7e4012 not found: ID does not exist" Oct 30 06:57:33.276489 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.276461 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6"] Oct 30 06:57:33.288822 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.288797 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-zzkq6"] Oct 30 06:57:33.339075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:33.339043 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" path="/var/lib/kubelet/pods/8505f718-e49c-4231-91f0-20dd4a1be595/volumes" Oct 30 06:57:33.701221 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:33.701191 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:57:34.173688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.173618 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:57:34.173688 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.173678 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:57:34.305417 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.305389 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2"] Oct 30 06:57:34.342086 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.342057 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx"] Oct 30 06:57:34.895946 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.895160 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:57:34.985321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.984440 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-certs\") pod \"dd098f33-cfd7-4505-8abd-0100edc4feef\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " Oct 30 06:57:34.985321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.984754 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-certs" pod "dd098f33-cfd7-4505-8abd-0100edc4feef" (UID: "dd098f33-cfd7-4505-8abd-0100edc4feef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:34.985321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.984828 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-envoy\") pod \"dd098f33-cfd7-4505-8abd-0100edc4feef\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " Oct 30 06:57:34.985321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.984854 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd098f33-cfd7-4505-8abd-0100edc4feef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd098f33-cfd7-4505-8abd-0100edc4feef-istiod-ca-cert\") pod \"dd098f33-cfd7-4505-8abd-0100edc4feef\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " Oct 30 06:57:34.985321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.984900 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-token\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-token\") pod \"dd098f33-cfd7-4505-8abd-0100edc4feef\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " Oct 30 06:57:34.985321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.984973 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-data\") pod \"dd098f33-cfd7-4505-8abd-0100edc4feef\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " Oct 30 06:57:34.985321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.985002 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-credential-socket\") pod \"dd098f33-cfd7-4505-8abd-0100edc4feef\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " Oct 30 06:57:34.985321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.985023 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-podinfo\") pod \"dd098f33-cfd7-4505-8abd-0100edc4feef\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " Oct 30 06:57:34.985321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.985061 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-kube-api-access-f9vqk\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-kube-api-access-f9vqk\") pod \"dd098f33-cfd7-4505-8abd-0100edc4feef\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " Oct 30 06:57:34.986586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.986372 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-data" pod "dd098f33-cfd7-4505-8abd-0100edc4feef" (UID: "dd098f33-cfd7-4505-8abd-0100edc4feef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:34.986586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.986412 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd098f33-cfd7-4505-8abd-0100edc4feef-istiod-ca-cert" pod "dd098f33-cfd7-4505-8abd-0100edc4feef" (UID: "dd098f33-cfd7-4505-8abd-0100edc4feef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:34.986586 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.986469 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-socket\") pod \"dd098f33-cfd7-4505-8abd-0100edc4feef\" (UID: \"dd098f33-cfd7-4505-8abd-0100edc4feef\") " Oct 30 06:57:34.986856 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.986699 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-socket" pod "dd098f33-cfd7-4505-8abd-0100edc4feef" (UID: "dd098f33-cfd7-4505-8abd-0100edc4feef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:34.987340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.986996 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-credential-socket" pod "dd098f33-cfd7-4505-8abd-0100edc4feef" (UID: "dd098f33-cfd7-4505-8abd-0100edc4feef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:34.987340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.987241 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.987340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.987262 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd098f33-cfd7-4505-8abd-0100edc4feef-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.987340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.987274 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.987340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.987285 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.987340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.987316 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.995707 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.994783 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-podinfo" pod "dd098f33-cfd7-4505-8abd-0100edc4feef" (UID: "dd098f33-cfd7-4505-8abd-0100edc4feef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:57:34.996289 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.996266 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-kube-api-access-f9vqk" pod "dd098f33-cfd7-4505-8abd-0100edc4feef" (UID: "dd098f33-cfd7-4505-8abd-0100edc4feef"). InnerVolumeSpecName "kube-api-access-f9vqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:35.000317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.996902 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-token" pod "dd098f33-cfd7-4505-8abd-0100edc4feef" (UID: "dd098f33-cfd7-4505-8abd-0100edc4feef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:35.000317 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:34.998402 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-envoy" pod "dd098f33-cfd7-4505-8abd-0100edc4feef" (UID: "dd098f33-cfd7-4505-8abd-0100edc4feef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:35.088282 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.088253 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:35.088282 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.088279 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:35.088282 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.088288 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd098f33-cfd7-4505-8abd-0100edc4feef-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:35.088534 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.088297 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9vqk\" (UniqueName: \"kubernetes.io/projected/dd098f33-cfd7-4505-8abd-0100edc4feef-kube-api-access-f9vqk\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:35.229255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.229225 4431 generic.go:356] "Generic (PLEG): container finished" podID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerID="e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7" exitCode=0 Oct 30 06:57:35.229515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.229276 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" Oct 30 06:57:35.229515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.229293 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" event={"ID":"dd098f33-cfd7-4505-8abd-0100edc4feef","Type":"ContainerDied","Data":"e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7"} Oct 30 06:57:35.229515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.229346 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt" event={"ID":"dd098f33-cfd7-4505-8abd-0100edc4feef","Type":"ContainerDied","Data":"503410b054f4f54772ebf1b2e056b38f029a173cf83b9d2ce7cebbb7f7528abd"} Oct 30 06:57:35.229515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.229368 4431 scope.go:122] "RemoveContainer" containerID="e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7" Oct 30 06:57:35.239990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.239970 4431 scope.go:122] "RemoveContainer" containerID="e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7" Oct 30 06:57:35.240243 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:35.240222 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7\": container with ID starting with e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7 not found: ID does not exist" containerID="e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7" Oct 30 06:57:35.240311 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.240250 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7"} err="failed to get container status \"e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7\": rpc error: code = NotFound desc = could not find container \"e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7\": container with ID starting with e58c72a1939ab080eb1e511919bf72821d581b54e74356b1b99761dca57f62c7 not found: ID does not exist" Oct 30 06:57:35.281370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.281343 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt"] Oct 30 06:57:35.293574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.293550 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-x8tzt"] Oct 30 06:57:35.350647 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:35.350602 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" path="/var/lib/kubelet/pods/dd098f33-cfd7-4505-8abd-0100edc4feef/volumes" Oct 30 06:57:39.306110 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:39.306067 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" podUID="918cfec5-83fd-469d-8a53-63990ff9376a" containerName="payload-processing" containerID="cri-o://a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9" gracePeriod=25 Oct 30 06:57:39.343382 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:39.343347 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" podUID="7376eddd-dd2f-4c7a-af89-7c9fb70abf70" containerName="payload-pre-processing" containerID="cri-o://ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968" gracePeriod=25 Oct 30 06:57:39.589769 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:39.589741 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:57:39.642658 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:39.642630 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918cfec5-83fd-469d-8a53-63990ff9376a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918cfec5-83fd-469d-8a53-63990ff9376a-plugins-config-volume\") pod \"918cfec5-83fd-469d-8a53-63990ff9376a\" (UID: \"918cfec5-83fd-469d-8a53-63990ff9376a\") " Oct 30 06:57:39.642828 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:39.642765 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918cfec5-83fd-469d-8a53-63990ff9376a-kube-api-access-gjwlv\" (UniqueName: \"kubernetes.io/projected/918cfec5-83fd-469d-8a53-63990ff9376a-kube-api-access-gjwlv\") pod \"918cfec5-83fd-469d-8a53-63990ff9376a\" (UID: \"918cfec5-83fd-469d-8a53-63990ff9376a\") " Oct 30 06:57:39.643137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:39.643110 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918cfec5-83fd-469d-8a53-63990ff9376a-plugins-config-volume" pod "918cfec5-83fd-469d-8a53-63990ff9376a" (UID: "918cfec5-83fd-469d-8a53-63990ff9376a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:39.643412 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:39.643317 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918cfec5-83fd-469d-8a53-63990ff9376a-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:39.645251 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:39.645149 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918cfec5-83fd-469d-8a53-63990ff9376a-kube-api-access-gjwlv" pod "918cfec5-83fd-469d-8a53-63990ff9376a" (UID: "918cfec5-83fd-469d-8a53-63990ff9376a"). InnerVolumeSpecName "kube-api-access-gjwlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:39.744258 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:39.744229 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjwlv\" (UniqueName: \"kubernetes.io/projected/918cfec5-83fd-469d-8a53-63990ff9376a-kube-api-access-gjwlv\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:40.263092 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.263059 4431 generic.go:356] "Generic (PLEG): container finished" podID="918cfec5-83fd-469d-8a53-63990ff9376a" containerID="a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9" exitCode=0 Oct 30 06:57:40.263279 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.263106 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" Oct 30 06:57:40.263279 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.263116 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" event={"ID":"918cfec5-83fd-469d-8a53-63990ff9376a","Type":"ContainerDied","Data":"a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9"} Oct 30 06:57:40.263279 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.263154 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2" event={"ID":"918cfec5-83fd-469d-8a53-63990ff9376a","Type":"ContainerDied","Data":"123f4728bfd8b6f696ccb98aba12f8ce68cdfda2461ef2a5b09e09b402d6646a"} Oct 30 06:57:40.263279 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.263177 4431 scope.go:122] "RemoveContainer" containerID="a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9" Oct 30 06:57:40.273206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.273191 4431 scope.go:122] "RemoveContainer" containerID="a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9" Oct 30 06:57:40.273457 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:40.273440 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9\": container with ID starting with a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9 not found: ID does not exist" containerID="a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9" Oct 30 06:57:40.273512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.273463 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9"} err="failed to get container status \"a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9\": rpc error: code = NotFound desc = could not find container \"a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9\": container with ID starting with a4c7abf6f59aebc55a5780e8655341a4c966699c726f458baa332e3cf2f023c9 not found: ID does not exist" Oct 30 06:57:40.315268 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.315231 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2"] Oct 30 06:57:40.325978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.325950 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5f459-77df44f7fd-zg4q2"] Oct 30 06:57:40.934466 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.934430 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9"] Oct 30 06:57:40.934764 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:40.934730 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" containerID="cri-o://7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819" gracePeriod=30 Oct 30 06:57:41.338589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:41.338555 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918cfec5-83fd-469d-8a53-63990ff9376a" path="/var/lib/kubelet/pods/918cfec5-83fd-469d-8a53-63990ff9376a/volumes" Oct 30 06:57:45.233454 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:45.233413 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" podUID="7376eddd-dd2f-4c7a-af89-7c9fb70abf70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:46.195337 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.195291 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:57:46.303066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.302987 4431 generic.go:356] "Generic (PLEG): container finished" podID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerID="7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819" exitCode=0 Oct 30 06:57:46.303066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.303035 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" Oct 30 06:57:46.303496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.303068 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" event={"ID":"54aa0ffd-3489-4d17-9171-f56a8664045c","Type":"ContainerDied","Data":"7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819"} Oct 30 06:57:46.303496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.303146 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9" event={"ID":"54aa0ffd-3489-4d17-9171-f56a8664045c","Type":"ContainerDied","Data":"fdc6b0a98acf6fbb4e4b080be76206a2cd717b3b5cc4ebe2f51de8726bde1f00"} Oct 30 06:57:46.303496 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.303176 4431 scope.go:122] "RemoveContainer" containerID="7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819" Oct 30 06:57:46.314106 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.313981 4431 scope.go:122] "RemoveContainer" containerID="1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e" Oct 30 06:57:46.324582 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.324565 4431 scope.go:122] "RemoveContainer" containerID="7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819" Oct 30 06:57:46.324862 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:46.324844 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819\": container with ID starting with 7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819 not found: ID does not exist" containerID="7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819" Oct 30 06:57:46.324917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.324870 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819"} err="failed to get container status \"7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819\": rpc error: code = NotFound desc = could not find container \"7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819\": container with ID starting with 7eb79505efc81f2e3fadf86e3399d4605ba365ca8f60cec93a5ffc2228147819 not found: ID does not exist" Oct 30 06:57:46.324917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.324886 4431 scope.go:122] "RemoveContainer" containerID="1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e" Oct 30 06:57:46.325157 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:57:46.325140 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e\": container with ID starting with 1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e not found: ID does not exist" containerID="1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e" Oct 30 06:57:46.325206 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.325164 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e"} err="failed to get container status \"1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e\": rpc error: code = NotFound desc = could not find container \"1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e\": container with ID starting with 1dfc06ef0456aa36335731b2f3136c91858a0d3822f51635e8541d94400c0e8e not found: ID does not exist" Oct 30 06:57:46.333738 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.333708 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-token\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-token\") pod \"54aa0ffd-3489-4d17-9171-f56a8664045c\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " Oct 30 06:57:46.333797 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.333752 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-data\") pod \"54aa0ffd-3489-4d17-9171-f56a8664045c\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " Oct 30 06:57:46.333841 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.333800 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-envoy\") pod \"54aa0ffd-3489-4d17-9171-f56a8664045c\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " Oct 30 06:57:46.333940 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.333925 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-certs\") pod \"54aa0ffd-3489-4d17-9171-f56a8664045c\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " Oct 30 06:57:46.333990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.333959 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-credential-socket\") pod \"54aa0ffd-3489-4d17-9171-f56a8664045c\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " Oct 30 06:57:46.334039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.333997 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-podinfo\") pod \"54aa0ffd-3489-4d17-9171-f56a8664045c\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " Oct 30 06:57:46.334090 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334056 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-data" pod "54aa0ffd-3489-4d17-9171-f56a8664045c" (UID: "54aa0ffd-3489-4d17-9171-f56a8664045c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:46.334145 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334120 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-kube-api-access-vmhjm\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-kube-api-access-vmhjm\") pod \"54aa0ffd-3489-4d17-9171-f56a8664045c\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " Oct 30 06:57:46.334200 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334185 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54aa0ffd-3489-4d17-9171-f56a8664045c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54aa0ffd-3489-4d17-9171-f56a8664045c-istiod-ca-cert\") pod \"54aa0ffd-3489-4d17-9171-f56a8664045c\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " Oct 30 06:57:46.334200 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334176 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-certs" pod "54aa0ffd-3489-4d17-9171-f56a8664045c" (UID: "54aa0ffd-3489-4d17-9171-f56a8664045c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:46.334340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334213 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-socket\") pod \"54aa0ffd-3489-4d17-9171-f56a8664045c\" (UID: \"54aa0ffd-3489-4d17-9171-f56a8664045c\") " Oct 30 06:57:46.334340 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334258 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-credential-socket" pod "54aa0ffd-3489-4d17-9171-f56a8664045c" (UID: "54aa0ffd-3489-4d17-9171-f56a8664045c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:46.334556 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334531 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-socket" pod "54aa0ffd-3489-4d17-9171-f56a8664045c" (UID: "54aa0ffd-3489-4d17-9171-f56a8664045c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:46.334652 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334593 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54aa0ffd-3489-4d17-9171-f56a8664045c-istiod-ca-cert" pod "54aa0ffd-3489-4d17-9171-f56a8664045c" (UID: "54aa0ffd-3489-4d17-9171-f56a8664045c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:46.334789 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334768 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:46.334853 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334797 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:46.334853 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334812 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54aa0ffd-3489-4d17-9171-f56a8664045c-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:46.334853 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334826 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:46.334853 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.334839 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:46.336041 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.336020 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-token" pod "54aa0ffd-3489-4d17-9171-f56a8664045c" (UID: "54aa0ffd-3489-4d17-9171-f56a8664045c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:46.336134 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.336049 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-envoy" pod "54aa0ffd-3489-4d17-9171-f56a8664045c" (UID: "54aa0ffd-3489-4d17-9171-f56a8664045c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:46.336227 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.336200 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-podinfo" pod "54aa0ffd-3489-4d17-9171-f56a8664045c" (UID: "54aa0ffd-3489-4d17-9171-f56a8664045c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:57:46.336339 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.336264 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-kube-api-access-vmhjm" pod "54aa0ffd-3489-4d17-9171-f56a8664045c" (UID: "54aa0ffd-3489-4d17-9171-f56a8664045c"). InnerVolumeSpecName "kube-api-access-vmhjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:46.422325 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.422289 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf"] Oct 30 06:57:46.432433 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.432404 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6"] Oct 30 06:57:46.435906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.435750 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmhjm\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-kube-api-access-vmhjm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:46.436066 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.436040 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:46.436168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.436073 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:46.436168 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.436088 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54aa0ffd-3489-4d17-9171-f56a8664045c-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:46.649347 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.649314 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9"] Oct 30 06:57:46.655259 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:46.655232 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-b9ff93-openshift-default-7fb7b8684c-jmrz9"] Oct 30 06:57:47.337161 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:47.337129 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" path="/var/lib/kubelet/pods/54aa0ffd-3489-4d17-9171-f56a8664045c/volumes" Oct 30 06:57:52.156649 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:52.156614 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7"] Oct 30 06:57:55.233675 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:55.233639 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" podUID="7376eddd-dd2f-4c7a-af89-7c9fb70abf70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:57:55.702209 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:55.702180 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr"] Oct 30 06:57:57.157182 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.157136 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" podUID="96a04f24-d9ec-472f-8ec2-002b5cabf3da" containerName="payload-processing" containerID="cri-o://5bb8a6e73658cb32b9c71b66b3ffd28669e32c0bca0a9886abad3e96d6e16321" gracePeriod=25 Oct 30 06:57:57.381714 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.381672 4431 generic.go:356] "Generic (PLEG): container finished" podID="96a04f24-d9ec-472f-8ec2-002b5cabf3da" containerID="5bb8a6e73658cb32b9c71b66b3ffd28669e32c0bca0a9886abad3e96d6e16321" exitCode=0 Oct 30 06:57:57.381895 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.381739 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" event={"ID":"96a04f24-d9ec-472f-8ec2-002b5cabf3da","Type":"ContainerDied","Data":"5bb8a6e73658cb32b9c71b66b3ffd28669e32c0bca0a9886abad3e96d6e16321"} Oct 30 06:57:57.428641 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.428614 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:57:57.564176 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.564144 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a04f24-d9ec-472f-8ec2-002b5cabf3da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a04f24-d9ec-472f-8ec2-002b5cabf3da-plugins-config-volume\") pod \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\" (UID: \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\") " Oct 30 06:57:57.564344 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.564280 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a04f24-d9ec-472f-8ec2-002b5cabf3da-kube-api-access-xz7xd\" (UniqueName: \"kubernetes.io/projected/96a04f24-d9ec-472f-8ec2-002b5cabf3da-kube-api-access-xz7xd\") pod \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\" (UID: \"96a04f24-d9ec-472f-8ec2-002b5cabf3da\") " Oct 30 06:57:57.564615 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.564591 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a04f24-d9ec-472f-8ec2-002b5cabf3da-plugins-config-volume" pod "96a04f24-d9ec-472f-8ec2-002b5cabf3da" (UID: "96a04f24-d9ec-472f-8ec2-002b5cabf3da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:57.566297 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.566272 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a04f24-d9ec-472f-8ec2-002b5cabf3da-kube-api-access-xz7xd" pod "96a04f24-d9ec-472f-8ec2-002b5cabf3da" (UID: "96a04f24-d9ec-472f-8ec2-002b5cabf3da"). InnerVolumeSpecName "kube-api-access-xz7xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:57.665423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.665359 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a04f24-d9ec-472f-8ec2-002b5cabf3da-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:57.665423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:57.665384 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz7xd\" (UniqueName: \"kubernetes.io/projected/96a04f24-d9ec-472f-8ec2-002b5cabf3da-kube-api-access-xz7xd\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:58.389461 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:58.389433 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" Oct 30 06:57:58.389891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:58.389478 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7" event={"ID":"96a04f24-d9ec-472f-8ec2-002b5cabf3da","Type":"ContainerDied","Data":"94603a0b7b3e7178b2107708f6259ba7c6ba60206ea1f85f1c7cf844eee51b8b"} Oct 30 06:57:58.389891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:58.389548 4431 scope.go:122] "RemoveContainer" containerID="5bb8a6e73658cb32b9c71b66b3ffd28669e32c0bca0a9886abad3e96d6e16321" Oct 30 06:57:58.440346 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:58.440319 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7"] Oct 30 06:57:58.452158 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:58.452137 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a519e9-8c464697d-gpjt7"] Oct 30 06:57:59.338655 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:59.338625 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a04f24-d9ec-472f-8ec2-002b5cabf3da" path="/var/lib/kubelet/pods/96a04f24-d9ec-472f-8ec2-002b5cabf3da/volumes" Oct 30 06:57:59.609362 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:59.609338 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:57:59.787734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:59.787701 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-kube-api-access-8s8r9\" (UniqueName: \"kubernetes.io/projected/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-kube-api-access-8s8r9\") pod \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\" (UID: \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\") " Oct 30 06:57:59.787901 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:59.787870 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-plugins-config-volume\") pod \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\" (UID: \"7376eddd-dd2f-4c7a-af89-7c9fb70abf70\") " Oct 30 06:57:59.788243 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:59.788218 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-plugins-config-volume" pod "7376eddd-dd2f-4c7a-af89-7c9fb70abf70" (UID: "7376eddd-dd2f-4c7a-af89-7c9fb70abf70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:59.789728 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:59.789702 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-kube-api-access-8s8r9" pod "7376eddd-dd2f-4c7a-af89-7c9fb70abf70" (UID: "7376eddd-dd2f-4c7a-af89-7c9fb70abf70"). InnerVolumeSpecName "kube-api-access-8s8r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:59.889469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:59.889411 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:57:59.889469 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:57:59.889437 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8r9\" (UniqueName: \"kubernetes.io/projected/7376eddd-dd2f-4c7a-af89-7c9fb70abf70-kube-api-access-8s8r9\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:00.166969 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.166880 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c"] Oct 30 06:58:00.167149 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.167072 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" podUID="3bdd9574-f2ab-449e-916e-74fd2f3cda2d" containerName="maas-api" containerID="cri-o://f4544dde7924ef435a1c381a7681abd90f4990de06c84676c9b5019c51773c2b" gracePeriod=30 Oct 30 06:58:00.423241 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.423127 4431 generic.go:356] "Generic (PLEG): container finished" podID="7376eddd-dd2f-4c7a-af89-7c9fb70abf70" containerID="ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968" exitCode=0 Oct 30 06:58:00.423435 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.423270 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" Oct 30 06:58:00.423761 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.423734 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" event={"ID":"7376eddd-dd2f-4c7a-af89-7c9fb70abf70","Type":"ContainerDied","Data":"ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968"} Oct 30 06:58:00.424083 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.423978 4431 scope.go:122] "RemoveContainer" containerID="ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968" Oct 30 06:58:00.424806 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.423932 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx" event={"ID":"7376eddd-dd2f-4c7a-af89-7c9fb70abf70","Type":"ContainerDied","Data":"7982f3eac5729b6b3a6dfb86670e3e66e1a19941d24121c2d603c5d3e2c7f684"} Oct 30 06:58:00.430075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.428606 4431 generic.go:356] "Generic (PLEG): container finished" podID="3bdd9574-f2ab-449e-916e-74fd2f3cda2d" containerID="f4544dde7924ef435a1c381a7681abd90f4990de06c84676c9b5019c51773c2b" exitCode=0 Oct 30 06:58:00.430075 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.428661 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" event={"ID":"3bdd9574-f2ab-449e-916e-74fd2f3cda2d","Type":"ContainerDied","Data":"f4544dde7924ef435a1c381a7681abd90f4990de06c84676c9b5019c51773c2b"} Oct 30 06:58:00.458795 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.458776 4431 scope.go:122] "RemoveContainer" containerID="ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968" Oct 30 06:58:00.459058 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:58:00.459037 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968\": container with ID starting with ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968 not found: ID does not exist" containerID="ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968" Oct 30 06:58:00.459125 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.459070 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968"} err="failed to get container status \"ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968\": rpc error: code = NotFound desc = could not find container \"ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968\": container with ID starting with ccd07b0367ef5372215413b0039adf892ab4272a37a53ef90d0c43d33158d968 not found: ID does not exist" Oct 30 06:58:00.463125 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.463102 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:58:00.493530 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.493505 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx"] Oct 30 06:58:00.501711 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.501690 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5f459-fc97c8bf9-9xqfx"] Oct 30 06:58:00.597749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.597710 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-openshift-service-ca\") pod \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " Oct 30 06:58:00.597953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.597784 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-kube-api-access-qcm6n\" (UniqueName: \"kubernetes.io/projected/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-kube-api-access-qcm6n\") pod \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " Oct 30 06:58:00.597953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.597824 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-maas-api-tls\") pod \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " Oct 30 06:58:00.597953 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.597945 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-metrics-certs\") pod \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\" (UID: \"3bdd9574-f2ab-449e-916e-74fd2f3cda2d\") " Oct 30 06:58:00.601748 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.601689 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-kube-api-access-qcm6n" pod "3bdd9574-f2ab-449e-916e-74fd2f3cda2d" (UID: "3bdd9574-f2ab-449e-916e-74fd2f3cda2d"). InnerVolumeSpecName "kube-api-access-qcm6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:00.602749 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.602720 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-metrics-certs" pod "3bdd9574-f2ab-449e-916e-74fd2f3cda2d" (UID: "3bdd9574-f2ab-449e-916e-74fd2f3cda2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:00.602878 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.602861 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-openshift-service-ca" pod "3bdd9574-f2ab-449e-916e-74fd2f3cda2d" (UID: "3bdd9574-f2ab-449e-916e-74fd2f3cda2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:00.605848 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.605820 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-maas-api-tls" pod "3bdd9574-f2ab-449e-916e-74fd2f3cda2d" (UID: "3bdd9574-f2ab-449e-916e-74fd2f3cda2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:00.699628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.699597 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:00.699628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.699625 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcm6n\" (UniqueName: \"kubernetes.io/projected/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-kube-api-access-qcm6n\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:00.700037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.699635 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:00.700037 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.699645 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bdd9574-f2ab-449e-916e-74fd2f3cda2d-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:00.703166 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:00.703129 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" podUID="95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" containerName="payload-processing" containerID="cri-o://fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49" gracePeriod=25 Oct 30 06:58:01.339063 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.339030 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7376eddd-dd2f-4c7a-af89-7c9fb70abf70" path="/var/lib/kubelet/pods/7376eddd-dd2f-4c7a-af89-7c9fb70abf70/volumes" Oct 30 06:58:01.438163 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.438134 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" Oct 30 06:58:01.438297 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.438158 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c" event={"ID":"3bdd9574-f2ab-449e-916e-74fd2f3cda2d","Type":"ContainerDied","Data":"09d5da8e23ff74776f391a8281d6d794adeda68afcba2ea1b57f340e807319f3"} Oct 30 06:58:01.438297 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.438217 4431 scope.go:122] "RemoveContainer" containerID="f4544dde7924ef435a1c381a7681abd90f4990de06c84676c9b5019c51773c2b" Oct 30 06:58:01.443293 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.443251 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" podUID="1d337a85-2994-4119-8d76-c0bbaca18d5e" containerName="main" containerID="cri-o://a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720" gracePeriod=45 Oct 30 06:58:01.452127 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.452099 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" containerID="cri-o://0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6" gracePeriod=45 Oct 30 06:58:01.489363 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.489324 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c"] Oct 30 06:58:01.501635 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.501612 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5f459-5f75bd8ccb-h6g5c"] Oct 30 06:58:01.719589 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.719555 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:58:01.747287 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.747267 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:58:01.811815 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.811780 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-tmp-dir\") pod \"1d337a85-2994-4119-8d76-c0bbaca18d5e\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " Oct 30 06:58:01.812015 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.811835 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-home\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-home\") pod \"eecc4014-0448-4d17-a646-acd40564b5cc\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " Oct 30 06:58:01.812015 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.811870 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-model-cache\") pod \"eecc4014-0448-4d17-a646-acd40564b5cc\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " Oct 30 06:58:01.812015 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.811902 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-home\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-home\") pod \"1d337a85-2994-4119-8d76-c0bbaca18d5e\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " Oct 30 06:58:01.812015 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.811955 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-dshm\") pod \"1d337a85-2994-4119-8d76-c0bbaca18d5e\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " Oct 30 06:58:01.812015 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812015 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d337a85-2994-4119-8d76-c0bbaca18d5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d337a85-2994-4119-8d76-c0bbaca18d5e-tls-certs\") pod \"1d337a85-2994-4119-8d76-c0bbaca18d5e\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " Oct 30 06:58:01.812329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812058 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-dshm\") pod \"eecc4014-0448-4d17-a646-acd40564b5cc\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " Oct 30 06:58:01.812329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812065 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-tmp-dir" pod "1d337a85-2994-4119-8d76-c0bbaca18d5e" (UID: "1d337a85-2994-4119-8d76-c0bbaca18d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.812329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812154 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d337a85-2994-4119-8d76-c0bbaca18d5e-kube-api-access-779qw\" (UniqueName: \"kubernetes.io/projected/1d337a85-2994-4119-8d76-c0bbaca18d5e-kube-api-access-779qw\") pod \"1d337a85-2994-4119-8d76-c0bbaca18d5e\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " Oct 30 06:58:01.812329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812213 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-tmp-dir\") pod \"eecc4014-0448-4d17-a646-acd40564b5cc\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " Oct 30 06:58:01.812329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812263 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-model-cache\") pod \"1d337a85-2994-4119-8d76-c0bbaca18d5e\" (UID: \"1d337a85-2994-4119-8d76-c0bbaca18d5e\") " Oct 30 06:58:01.812329 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812316 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eecc4014-0448-4d17-a646-acd40564b5cc-kube-api-access-nnlms\" (UniqueName: \"kubernetes.io/projected/eecc4014-0448-4d17-a646-acd40564b5cc-kube-api-access-nnlms\") pod \"eecc4014-0448-4d17-a646-acd40564b5cc\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " Oct 30 06:58:01.812670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812357 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eecc4014-0448-4d17-a646-acd40564b5cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/eecc4014-0448-4d17-a646-acd40564b5cc-tls-certs\") pod \"eecc4014-0448-4d17-a646-acd40564b5cc\" (UID: \"eecc4014-0448-4d17-a646-acd40564b5cc\") " Oct 30 06:58:01.812670 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812657 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-tmp-dir" pod "eecc4014-0448-4d17-a646-acd40564b5cc" (UID: "eecc4014-0448-4d17-a646-acd40564b5cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.812883 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812860 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.812963 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.812888 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.814132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.813097 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-model-cache" pod "1d337a85-2994-4119-8d76-c0bbaca18d5e" (UID: "1d337a85-2994-4119-8d76-c0bbaca18d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.814132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.813604 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-home" pod "1d337a85-2994-4119-8d76-c0bbaca18d5e" (UID: "1d337a85-2994-4119-8d76-c0bbaca18d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.814132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.813767 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-home" pod "eecc4014-0448-4d17-a646-acd40564b5cc" (UID: "eecc4014-0448-4d17-a646-acd40564b5cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.814132 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.813910 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-model-cache" pod "eecc4014-0448-4d17-a646-acd40564b5cc" (UID: "eecc4014-0448-4d17-a646-acd40564b5cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.815814 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.815758 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-dshm" pod "eecc4014-0448-4d17-a646-acd40564b5cc" (UID: "eecc4014-0448-4d17-a646-acd40564b5cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.816629 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.816602 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecc4014-0448-4d17-a646-acd40564b5cc-kube-api-access-nnlms" pod "eecc4014-0448-4d17-a646-acd40564b5cc" (UID: "eecc4014-0448-4d17-a646-acd40564b5cc"). InnerVolumeSpecName "kube-api-access-nnlms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:01.817638 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.817609 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d337a85-2994-4119-8d76-c0bbaca18d5e-kube-api-access-779qw" pod "1d337a85-2994-4119-8d76-c0bbaca18d5e" (UID: "1d337a85-2994-4119-8d76-c0bbaca18d5e"). InnerVolumeSpecName "kube-api-access-779qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:01.818148 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.818099 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eecc4014-0448-4d17-a646-acd40564b5cc-tls-certs" pod "eecc4014-0448-4d17-a646-acd40564b5cc" (UID: "eecc4014-0448-4d17-a646-acd40564b5cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:01.818982 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.818876 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d337a85-2994-4119-8d76-c0bbaca18d5e-tls-certs" pod "1d337a85-2994-4119-8d76-c0bbaca18d5e" (UID: "1d337a85-2994-4119-8d76-c0bbaca18d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:01.819330 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.819282 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-dshm" pod "1d337a85-2994-4119-8d76-c0bbaca18d5e" (UID: "1d337a85-2994-4119-8d76-c0bbaca18d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914246 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d337a85-2994-4119-8d76-c0bbaca18d5e-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914283 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914297 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779qw\" (UniqueName: \"kubernetes.io/projected/1d337a85-2994-4119-8d76-c0bbaca18d5e-kube-api-access-779qw\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914347 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914362 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnlms\" (UniqueName: \"kubernetes.io/projected/eecc4014-0448-4d17-a646-acd40564b5cc-kube-api-access-nnlms\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914378 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eecc4014-0448-4d17-a646-acd40564b5cc-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914392 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914408 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eecc4014-0448-4d17-a646-acd40564b5cc-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914425 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.915271 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:01.914443 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d337a85-2994-4119-8d76-c0bbaca18d5e-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.445178 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.445144 4431 generic.go:356] "Generic (PLEG): container finished" podID="eecc4014-0448-4d17-a646-acd40564b5cc" containerID="0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6" exitCode=0 Oct 30 06:58:02.445377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.445197 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" Oct 30 06:58:02.445377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.445238 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" event={"ID":"eecc4014-0448-4d17-a646-acd40564b5cc","Type":"ContainerDied","Data":"0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6"} Oct 30 06:58:02.445377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.445280 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6" event={"ID":"eecc4014-0448-4d17-a646-acd40564b5cc","Type":"ContainerDied","Data":"f4bbb698737819b426b9d59b1952cf48ec20c9b389baaaf9e7ae98782ca82217"} Oct 30 06:58:02.445377 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.445310 4431 scope.go:122] "RemoveContainer" containerID="0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6" Oct 30 06:58:02.447772 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.447747 4431 generic.go:356] "Generic (PLEG): container finished" podID="1d337a85-2994-4119-8d76-c0bbaca18d5e" containerID="a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720" exitCode=0 Oct 30 06:58:02.447906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.447811 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" Oct 30 06:58:02.447906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.447828 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" event={"ID":"1d337a85-2994-4119-8d76-c0bbaca18d5e","Type":"ContainerDied","Data":"a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720"} Oct 30 06:58:02.447906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.447856 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" event={"ID":"1d337a85-2994-4119-8d76-c0bbaca18d5e","Type":"ContainerDied","Data":"113eba858a06ae2bd849722602bfa425ea06f80b63dcd716cfa1648e5a4e88a3"} Oct 30 06:58:02.455891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.455871 4431 scope.go:122] "RemoveContainer" containerID="0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6" Oct 30 06:58:02.456171 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:58:02.456150 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6\": container with ID starting with 0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6 not found: ID does not exist" containerID="0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6" Oct 30 06:58:02.456262 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.456176 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6"} err="failed to get container status \"0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6\": rpc error: code = NotFound desc = could not find container \"0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6\": container with ID starting with 0cb2996f235ff6a89301cb95dda1c160182aa21c9a21ed34225152fc404274b6 not found: ID does not exist" Oct 30 06:58:02.456262 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.456194 4431 scope.go:122] "RemoveContainer" containerID="a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720" Oct 30 06:58:02.469085 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.469069 4431 scope.go:122] "RemoveContainer" containerID="a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720" Oct 30 06:58:02.469353 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:58:02.469338 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720\": container with ID starting with a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720 not found: ID does not exist" containerID="a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720" Oct 30 06:58:02.469400 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.469358 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720"} err="failed to get container status \"a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720\": rpc error: code = NotFound desc = could not find container \"a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720\": container with ID starting with a6b8aa2f6bec6b083ac84abd318d63c821b5b65766b7b699ff21da3ec286e720 not found: ID does not exist" Oct 30 06:58:02.511865 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.511834 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6"] Oct 30 06:58:02.523491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.523469 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-w0-a519e9-kserve-7488d69c8c-54xr6"] Oct 30 06:58:02.552456 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.552436 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf"] Oct 30 06:58:02.558644 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.558623 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf"] Oct 30 06:58:02.642816 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.642786 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/distinct-2-w0-a519e9-kserve-878db4dcb-nbrrf" podUID="1d337a85-2994-4119-8d76-c0bbaca18d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 06:58:02.986225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:02.986187 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" podUID="95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:58:03.337514 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:03.337436 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d337a85-2994-4119-8d76-c0bbaca18d5e" path="/var/lib/kubelet/pods/1d337a85-2994-4119-8d76-c0bbaca18d5e/volumes" Oct 30 06:58:03.337893 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:03.337879 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bdd9574-f2ab-449e-916e-74fd2f3cda2d" path="/var/lib/kubelet/pods/3bdd9574-f2ab-449e-916e-74fd2f3cda2d/volumes" Oct 30 06:58:03.338273 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:03.338261 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" path="/var/lib/kubelet/pods/eecc4014-0448-4d17-a646-acd40564b5cc/volumes" Oct 30 06:58:03.701064 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:58:03.701028 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:58:12.986607 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:12.986562 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" podUID="95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:58:15.628864 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:15.628828 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp"] Oct 30 06:58:20.629751 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:20.629693 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" podUID="e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc" containerName="payload-processing" containerID="cri-o://ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab" gracePeriod=25 Oct 30 06:58:20.966795 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:20.966764 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:58:21.000908 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.000886 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:58:21.036933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.036908 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-plugins-config-volume\") pod \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\" (UID: \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\") " Oct 30 06:58:21.037071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.037033 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-kube-api-access-fq9fl\" (UniqueName: \"kubernetes.io/projected/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-kube-api-access-fq9fl\") pod \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\" (UID: \"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc\") " Oct 30 06:58:21.037354 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.037330 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-plugins-config-volume" pod "e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc" (UID: "e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:21.037491 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.037470 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:21.039222 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.039195 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-kube-api-access-fq9fl" pod "e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc" (UID: "e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc"). InnerVolumeSpecName "kube-api-access-fq9fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:21.138577 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.138547 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-kube-api-access-x5thb\" (UniqueName: \"kubernetes.io/projected/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-kube-api-access-x5thb\") pod \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\" (UID: \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\") " Oct 30 06:58:21.138790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.138774 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-plugins-config-volume\") pod \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\" (UID: \"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb\") " Oct 30 06:58:21.139071 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.139054 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq9fl\" (UniqueName: \"kubernetes.io/projected/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc-kube-api-access-fq9fl\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:21.139150 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.139092 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-plugins-config-volume" pod "95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" (UID: "95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:21.140681 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.140662 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-kube-api-access-x5thb" pod "95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" (UID: "95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb"). InnerVolumeSpecName "kube-api-access-x5thb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:21.239889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.239865 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:21.239889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.239885 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5thb\" (UniqueName: \"kubernetes.io/projected/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb-kube-api-access-x5thb\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:21.553699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.553622 4431 generic.go:356] "Generic (PLEG): container finished" podID="e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc" containerID="ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab" exitCode=0 Oct 30 06:58:21.553699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.553670 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" Oct 30 06:58:21.553699 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.553675 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" event={"ID":"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc","Type":"ContainerDied","Data":"ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab"} Oct 30 06:58:21.553923 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.553727 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp" event={"ID":"e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc","Type":"ContainerDied","Data":"0ac59e3fa9d6ff8bf6b2e5fcdee0a38bb58ee7ff11357c42b62e776576890bd8"} Oct 30 06:58:21.553923 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.553749 4431 scope.go:122] "RemoveContainer" containerID="ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab" Oct 30 06:58:21.555178 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.555159 4431 generic.go:356] "Generic (PLEG): container finished" podID="95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" containerID="fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49" exitCode=0 Oct 30 06:58:21.555288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.555226 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" Oct 30 06:58:21.555288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.555245 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" event={"ID":"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb","Type":"ContainerDied","Data":"fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49"} Oct 30 06:58:21.555288 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.555285 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr" event={"ID":"95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb","Type":"ContainerDied","Data":"d50a96057901533512b68ff98db63d3f4e10dd6b117434aa87c829dd645bdfca"} Oct 30 06:58:21.563552 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.563529 4431 scope.go:122] "RemoveContainer" containerID="ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab" Oct 30 06:58:21.563832 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:58:21.563812 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab\": container with ID starting with ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab not found: ID does not exist" containerID="ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab" Oct 30 06:58:21.563889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.563840 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab"} err="failed to get container status \"ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab\": rpc error: code = NotFound desc = could not find container \"ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab\": container with ID starting with ea32a8f71207b8db45f211ef5fca9a4a2562530cbd0c4d9b83185d63d132daab not found: ID does not exist" Oct 30 06:58:21.563889 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.563857 4431 scope.go:122] "RemoveContainer" containerID="fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49" Oct 30 06:58:21.572791 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.572770 4431 scope.go:122] "RemoveContainer" containerID="fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49" Oct 30 06:58:21.573062 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:58:21.573040 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49\": container with ID starting with fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49 not found: ID does not exist" containerID="fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49" Oct 30 06:58:21.573122 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.573073 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49"} err="failed to get container status \"fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49\": rpc error: code = NotFound desc = could not find container \"fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49\": container with ID starting with fa85730261e20fece9a6248beffa09a1cd5e4bd9ec10a82f7d60a952cab69c49 not found: ID does not exist" Oct 30 06:58:21.604416 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.604396 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp"] Oct 30 06:58:21.616257 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.616235 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-be48b4-6f5ccbbf64-9rmpp"] Oct 30 06:58:21.645458 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.645440 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr"] Oct 30 06:58:21.651648 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:21.651627 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89f101-786b784bfb-sppfr"] Oct 30 06:58:23.336972 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:23.336941 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" path="/var/lib/kubelet/pods/95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb/volumes" Oct 30 06:58:23.337459 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:23.337377 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc" path="/var/lib/kubelet/pods/e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc/volumes" Oct 30 06:58:30.587594 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.587551 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k"] Oct 30 06:58:30.587990 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.587794 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" podUID="a2c503db-91d1-47a9-a755-72fa60146690" containerName="maas-api" containerID="cri-o://617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc" gracePeriod=30 Oct 30 06:58:30.689614 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.689573 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" podUID="a2c503db-91d1-47a9-a755-72fa60146690" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.55:8443/health\": dial tcp 10.133.0.55:8443: connect: connection refused" Oct 30 06:58:30.880203 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.879247 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:58:30.976371 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.976342 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-metrics-certs\") pod \"a2c503db-91d1-47a9-a755-72fa60146690\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " Oct 30 06:58:30.976563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.976415 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c503db-91d1-47a9-a755-72fa60146690-kube-api-access-6hn5h\" (UniqueName: \"kubernetes.io/projected/a2c503db-91d1-47a9-a755-72fa60146690-kube-api-access-6hn5h\") pod \"a2c503db-91d1-47a9-a755-72fa60146690\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " Oct 30 06:58:30.976563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.976435 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-maas-api-tls\") pod \"a2c503db-91d1-47a9-a755-72fa60146690\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " Oct 30 06:58:30.976563 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.976500 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c503db-91d1-47a9-a755-72fa60146690-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c503db-91d1-47a9-a755-72fa60146690-openshift-service-ca\") pod \"a2c503db-91d1-47a9-a755-72fa60146690\" (UID: \"a2c503db-91d1-47a9-a755-72fa60146690\") " Oct 30 06:58:30.976948 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.976915 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c503db-91d1-47a9-a755-72fa60146690-openshift-service-ca" pod "a2c503db-91d1-47a9-a755-72fa60146690" (UID: "a2c503db-91d1-47a9-a755-72fa60146690"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:30.978624 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.978600 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c503db-91d1-47a9-a755-72fa60146690-kube-api-access-6hn5h" pod "a2c503db-91d1-47a9-a755-72fa60146690" (UID: "a2c503db-91d1-47a9-a755-72fa60146690"). InnerVolumeSpecName "kube-api-access-6hn5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:30.978740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.978645 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-maas-api-tls" pod "a2c503db-91d1-47a9-a755-72fa60146690" (UID: "a2c503db-91d1-47a9-a755-72fa60146690"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:30.978740 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:30.978671 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-metrics-certs" pod "a2c503db-91d1-47a9-a755-72fa60146690" (UID: "a2c503db-91d1-47a9-a755-72fa60146690"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:31.078004 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.077971 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.078004 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.077999 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hn5h\" (UniqueName: \"kubernetes.io/projected/a2c503db-91d1-47a9-a755-72fa60146690-kube-api-access-6hn5h\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.078004 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.078008 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c503db-91d1-47a9-a755-72fa60146690-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.078234 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.078017 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c503db-91d1-47a9-a755-72fa60146690-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.621336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.621222 4431 generic.go:356] "Generic (PLEG): container finished" podID="a2c503db-91d1-47a9-a755-72fa60146690" containerID="617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc" exitCode=0 Oct 30 06:58:31.621336 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.621270 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" Oct 30 06:58:31.621891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.621333 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" event={"ID":"a2c503db-91d1-47a9-a755-72fa60146690","Type":"ContainerDied","Data":"617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc"} Oct 30 06:58:31.621891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.621403 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k" event={"ID":"a2c503db-91d1-47a9-a755-72fa60146690","Type":"ContainerDied","Data":"346604b7ac4a562b7f8b82b527f11b20d5122b7aaf8c5b5f7c37138ef4831d13"} Oct 30 06:58:31.621891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.621438 4431 scope.go:122] "RemoveContainer" containerID="617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc" Oct 30 06:58:31.630891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.630871 4431 scope.go:122] "RemoveContainer" containerID="617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc" Oct 30 06:58:31.631185 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:58:31.631157 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc\": container with ID starting with 617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc not found: ID does not exist" containerID="617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc" Oct 30 06:58:31.631255 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.631190 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc"} err="failed to get container status \"617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc\": rpc error: code = NotFound desc = could not find container \"617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc\": container with ID starting with 617f57be51b6940ecc7d858c8adda9612fe5c83d94c7ffb597f191ce112030dc not found: ID does not exist" Oct 30 06:58:31.703828 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.700801 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k"] Oct 30 06:58:31.711146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:31.711118 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a519e9-6b66fd4c58-k6s2k"] Oct 30 06:58:33.337342 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:33.337283 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c503db-91d1-47a9-a755-72fa60146690" path="/var/lib/kubelet/pods/a2c503db-91d1-47a9-a755-72fa60146690/volumes" Oct 30 06:58:33.701942 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:58:33.701906 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:58:45.730845 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:45.730810 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p"] Oct 30 06:58:45.731413 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:45.731052 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" podUID="71c07881-c3bc-48f0-aff4-a4f20e84f387" containerName="discovery" containerID="cri-o://d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c" gracePeriod=30 Oct 30 06:58:45.992221 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:45.992161 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:58:46.057164 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.057132 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-dns-cert\") pod \"71c07881-c3bc-48f0-aff4-a4f20e84f387\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " Oct 30 06:58:46.057382 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.057268 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c07881-c3bc-48f0-aff4-a4f20e84f387-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c07881-c3bc-48f0-aff4-a4f20e84f387-local-certs\") pod \"71c07881-c3bc-48f0-aff4-a4f20e84f387\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " Oct 30 06:58:46.057382 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.057333 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-ca-configmap\") pod \"71c07881-c3bc-48f0-aff4-a4f20e84f387\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " Oct 30 06:58:46.057511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.057412 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-cacerts\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-cacerts\") pod \"71c07881-c3bc-48f0-aff4-a4f20e84f387\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " Oct 30 06:58:46.057511 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.057469 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-kubeconfig\") pod \"71c07881-c3bc-48f0-aff4-a4f20e84f387\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " Oct 30 06:58:46.057628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.057548 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-kube-api-access-d7q67\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-kube-api-access-d7q67\") pod \"71c07881-c3bc-48f0-aff4-a4f20e84f387\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " Oct 30 06:58:46.057628 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.057592 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-token\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-token\") pod \"71c07881-c3bc-48f0-aff4-a4f20e84f387\" (UID: \"71c07881-c3bc-48f0-aff4-a4f20e84f387\") " Oct 30 06:58:46.057839 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.057677 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-ca-configmap" pod "71c07881-c3bc-48f0-aff4-a4f20e84f387" (UID: "71c07881-c3bc-48f0-aff4-a4f20e84f387"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:46.058104 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.058081 4431 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-ca-configmap\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.060240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.060212 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c07881-c3bc-48f0-aff4-a4f20e84f387-local-certs" pod "71c07881-c3bc-48f0-aff4-a4f20e84f387" (UID: "71c07881-c3bc-48f0-aff4-a4f20e84f387"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:46.060240 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.060215 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-kubeconfig" pod "71c07881-c3bc-48f0-aff4-a4f20e84f387" (UID: "71c07881-c3bc-48f0-aff4-a4f20e84f387"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:46.060510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.060239 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-cacerts" pod "71c07881-c3bc-48f0-aff4-a4f20e84f387" (UID: "71c07881-c3bc-48f0-aff4-a4f20e84f387"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:46.060510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.060259 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-kube-api-access-d7q67" pod "71c07881-c3bc-48f0-aff4-a4f20e84f387" (UID: "71c07881-c3bc-48f0-aff4-a4f20e84f387"). InnerVolumeSpecName "kube-api-access-d7q67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:46.060632 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.060613 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-dns-cert" pod "71c07881-c3bc-48f0-aff4-a4f20e84f387" (UID: "71c07881-c3bc-48f0-aff4-a4f20e84f387"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:46.061119 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.061099 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-token" pod "71c07881-c3bc-48f0-aff4-a4f20e84f387" (UID: "71c07881-c3bc-48f0-aff4-a4f20e84f387"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:46.159094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.159061 4431 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-cacerts\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159094 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.159093 4431 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-kubeconfig\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.159108 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7q67\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-kube-api-access-d7q67\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.159123 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.159135 4431 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c07881-c3bc-48f0-aff4-a4f20e84f387-istio-csr-dns-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.159319 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.159147 4431 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c07881-c3bc-48f0-aff4-a4f20e84f387-local-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.702770 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.702733 4431 generic.go:356] "Generic (PLEG): container finished" podID="71c07881-c3bc-48f0-aff4-a4f20e84f387" containerID="d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c" exitCode=0 Oct 30 06:58:46.702933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.702798 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" Oct 30 06:58:46.702933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.702816 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" event={"ID":"71c07881-c3bc-48f0-aff4-a4f20e84f387","Type":"ContainerDied","Data":"d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c"} Oct 30 06:58:46.702933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.702893 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p" event={"ID":"71c07881-c3bc-48f0-aff4-a4f20e84f387","Type":"ContainerDied","Data":"a6171871836c387681ad60e0bfb00ba61a30449cfe6fe1915bce9c803b728699"} Oct 30 06:58:46.702933 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.702915 4431 scope.go:122] "RemoveContainer" containerID="d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c" Oct 30 06:58:46.720691 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.720673 4431 scope.go:122] "RemoveContainer" containerID="d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c" Oct 30 06:58:46.720930 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:58:46.720908 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c\": container with ID starting with d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c not found: ID does not exist" containerID="d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c" Oct 30 06:58:46.720978 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.720937 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c"} err="failed to get container status \"d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c\": rpc error: code = NotFound desc = could not find container \"d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c\": container with ID starting with d00d19a3929e6bd395e10003deca91429aa19428db3beeba5e6f84c4da44696c not found: ID does not exist" Oct 30 06:58:46.766106 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.766079 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p"] Oct 30 06:58:46.771744 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:46.771720 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f5g5p"] Oct 30 06:58:47.337189 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:47.337160 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c07881-c3bc-48f0-aff4-a4f20e84f387" path="/var/lib/kubelet/pods/71c07881-c3bc-48f0-aff4-a4f20e84f387/volumes" Oct 30 06:58:51.778915 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:51.778872 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr"] Oct 30 06:58:56.779372 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:58:56.779334 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" podUID="03aa02fa-ef29-4efb-ae52-5ed210c4a404" containerName="payload-pre-processing" containerID="cri-o://cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80" gracePeriod=25 Oct 30 06:59:00.636547 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:00.636510 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk"] Oct 30 06:59:00.637012 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:00.636772 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" containerID="cri-o://7a3840710cab03c3d9d7ee151d3dafbe3b8689bd7c3ffeca3f5a9e13c1cfc1c4" gracePeriod=30 Oct 30 06:59:02.974140 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:02.974093 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" podUID="03aa02fa-ef29-4efb-ae52-5ed210c4a404" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:03.701772 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:59:03.701731 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:59:05.805810 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.805776 4431 generic.go:356] "Generic (PLEG): container finished" podID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerID="7a3840710cab03c3d9d7ee151d3dafbe3b8689bd7c3ffeca3f5a9e13c1cfc1c4" exitCode=0 Oct 30 06:59:05.806195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.805854 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" event={"ID":"1db731cb-7cbb-4aef-9ef2-820c21f6317d","Type":"ContainerDied","Data":"7a3840710cab03c3d9d7ee151d3dafbe3b8689bd7c3ffeca3f5a9e13c1cfc1c4"} Oct 30 06:59:05.806195 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.805934 4431 scope.go:122] "RemoveContainer" containerID="5d8ad6d08592b2fd578051ccb77223c3bde9552ecc1234885b937ee9abb7ac14" Oct 30 06:59:05.890854 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.890832 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:59:05.964671 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.964640 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-kube-api-access-4xdtf\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-kube-api-access-4xdtf\") pod \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " Oct 30 06:59:05.964809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.964727 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-socket\") pod \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " Oct 30 06:59:05.964809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.964757 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-credential-socket\") pod \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " Oct 30 06:59:05.964809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.964788 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-envoy\") pod \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " Oct 30 06:59:05.964938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.964855 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-token\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-token\") pod \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " Oct 30 06:59:05.964938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.964886 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-podinfo\") pod \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " Oct 30 06:59:05.964938 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.964915 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-certs\") pod \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " Oct 30 06:59:05.965039 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.964975 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istiod-ca-cert\") pod \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " Oct 30 06:59:05.965080 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.965046 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-data\") pod \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\" (UID: \"1db731cb-7cbb-4aef-9ef2-820c21f6317d\") " Oct 30 06:59:05.965790 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.965766 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-data" pod "1db731cb-7cbb-4aef-9ef2-820c21f6317d" (UID: "1db731cb-7cbb-4aef-9ef2-820c21f6317d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:05.966859 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.966830 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-socket" pod "1db731cb-7cbb-4aef-9ef2-820c21f6317d" (UID: "1db731cb-7cbb-4aef-9ef2-820c21f6317d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:05.967046 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.967025 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-credential-socket" pod "1db731cb-7cbb-4aef-9ef2-820c21f6317d" (UID: "1db731cb-7cbb-4aef-9ef2-820c21f6317d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:05.968160 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.968132 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-kube-api-access-4xdtf" pod "1db731cb-7cbb-4aef-9ef2-820c21f6317d" (UID: "1db731cb-7cbb-4aef-9ef2-820c21f6317d"). InnerVolumeSpecName "kube-api-access-4xdtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:05.968423 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.968398 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-certs" pod "1db731cb-7cbb-4aef-9ef2-820c21f6317d" (UID: "1db731cb-7cbb-4aef-9ef2-820c21f6317d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:05.968868 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.968845 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istiod-ca-cert" pod "1db731cb-7cbb-4aef-9ef2-820c21f6317d" (UID: "1db731cb-7cbb-4aef-9ef2-820c21f6317d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:05.969618 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.969593 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-token" pod "1db731cb-7cbb-4aef-9ef2-820c21f6317d" (UID: "1db731cb-7cbb-4aef-9ef2-820c21f6317d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:05.970027 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.970011 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-envoy" pod "1db731cb-7cbb-4aef-9ef2-820c21f6317d" (UID: "1db731cb-7cbb-4aef-9ef2-820c21f6317d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:05.970361 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:05.970346 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-podinfo" pod "1db731cb-7cbb-4aef-9ef2-820c21f6317d" (UID: "1db731cb-7cbb-4aef-9ef2-820c21f6317d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:59:06.066515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.066490 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:06.066515 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.066512 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xdtf\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-kube-api-access-4xdtf\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:06.066653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.066522 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:06.066653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.066530 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:06.066653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.066540 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:06.066653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.066547 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:06.066653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.066555 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:06.066653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.066562 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db731cb-7cbb-4aef-9ef2-820c21f6317d-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:06.066653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.066571 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db731cb-7cbb-4aef-9ef2-820c21f6317d-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:06.813387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.813349 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" event={"ID":"1db731cb-7cbb-4aef-9ef2-820c21f6317d","Type":"ContainerDied","Data":"8fb77764a51c738e123e1a472ce7a5e5ff6eef91afbef5ed44cec4ce8bf0737b"} Oct 30 06:59:06.813387 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.813382 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk" Oct 30 06:59:06.813886 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.813422 4431 scope.go:122] "RemoveContainer" containerID="7a3840710cab03c3d9d7ee151d3dafbe3b8689bd7c3ffeca3f5a9e13c1cfc1c4" Oct 30 06:59:06.877717 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.877681 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk"] Oct 30 06:59:06.889951 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:06.889921 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b8ca49-openshift-default-7d467d9545-njslk"] Oct 30 06:59:07.337321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:07.337265 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" path="/var/lib/kubelet/pods/1db731cb-7cbb-4aef-9ef2-820c21f6317d/volumes" Oct 30 06:59:11.695713 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:11.695680 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt"] Oct 30 06:59:11.728113 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:11.728081 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl"] Oct 30 06:59:12.973891 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:12.973859 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" podUID="03aa02fa-ef29-4efb-ae52-5ed210c4a404" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:13.127932 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:13.127834 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz"] Oct 30 06:59:13.128141 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:13.128097 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" containerID="cri-o://ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee" gracePeriod=30 Oct 30 06:59:16.696444 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:16.696403 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" podUID="f5b9d11a-e96d-4733-b920-2ecf04bee128" containerName="payload-processing" containerID="cri-o://48bbca1a8d749e10307bbbc3858d2c478da536b89c4584326d7a7674f3c103e3" gracePeriod=25 Oct 30 06:59:16.728712 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:16.728682 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" podUID="f8efc049-5c75-4627-b33c-b0407d1a2067" containerName="payload-pre-processing" containerID="cri-o://e07b7acb537e23ce9d5a725f52d573448803ecceac78666421af141584755bd3" gracePeriod=25 Oct 30 06:59:16.866225 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:16.866198 4431 generic.go:356] "Generic (PLEG): container finished" podID="f5b9d11a-e96d-4733-b920-2ecf04bee128" containerID="48bbca1a8d749e10307bbbc3858d2c478da536b89c4584326d7a7674f3c103e3" exitCode=0 Oct 30 06:59:16.866379 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:16.866282 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" event={"ID":"f5b9d11a-e96d-4733-b920-2ecf04bee128","Type":"ContainerDied","Data":"48bbca1a8d749e10307bbbc3858d2c478da536b89c4584326d7a7674f3c103e3"} Oct 30 06:59:16.867962 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:16.867939 4431 generic.go:356] "Generic (PLEG): container finished" podID="f8efc049-5c75-4627-b33c-b0407d1a2067" containerID="e07b7acb537e23ce9d5a725f52d573448803ecceac78666421af141584755bd3" exitCode=0 Oct 30 06:59:16.868065 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:16.868016 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" event={"ID":"f8efc049-5c75-4627-b33c-b0407d1a2067","Type":"ContainerDied","Data":"e07b7acb537e23ce9d5a725f52d573448803ecceac78666421af141584755bd3"} Oct 30 06:59:17.069818 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.069788 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:59:17.072057 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.072028 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:59:17.093592 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.093571 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:59:17.181380 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.181353 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b9d11a-e96d-4733-b920-2ecf04bee128-kube-api-access-xncj8\" (UniqueName: \"kubernetes.io/projected/f5b9d11a-e96d-4733-b920-2ecf04bee128-kube-api-access-xncj8\") pod \"f5b9d11a-e96d-4733-b920-2ecf04bee128\" (UID: \"f5b9d11a-e96d-4733-b920-2ecf04bee128\") " Oct 30 06:59:17.181578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.181393 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8efc049-5c75-4627-b33c-b0407d1a2067-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8efc049-5c75-4627-b33c-b0407d1a2067-plugins-config-volume\") pod \"f8efc049-5c75-4627-b33c-b0407d1a2067\" (UID: \"f8efc049-5c75-4627-b33c-b0407d1a2067\") " Oct 30 06:59:17.181578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.181434 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03aa02fa-ef29-4efb-ae52-5ed210c4a404-kube-api-access-8zvzv\" (UniqueName: \"kubernetes.io/projected/03aa02fa-ef29-4efb-ae52-5ed210c4a404-kube-api-access-8zvzv\") pod \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\" (UID: \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\") " Oct 30 06:59:17.181578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.181516 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b9d11a-e96d-4733-b920-2ecf04bee128-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b9d11a-e96d-4733-b920-2ecf04bee128-plugins-config-volume\") pod \"f5b9d11a-e96d-4733-b920-2ecf04bee128\" (UID: \"f5b9d11a-e96d-4733-b920-2ecf04bee128\") " Oct 30 06:59:17.181578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.181549 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03aa02fa-ef29-4efb-ae52-5ed210c4a404-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa02fa-ef29-4efb-ae52-5ed210c4a404-plugins-config-volume\") pod \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\" (UID: \"03aa02fa-ef29-4efb-ae52-5ed210c4a404\") " Oct 30 06:59:17.181578 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.181577 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8efc049-5c75-4627-b33c-b0407d1a2067-kube-api-access-2wvzr\" (UniqueName: \"kubernetes.io/projected/f8efc049-5c75-4627-b33c-b0407d1a2067-kube-api-access-2wvzr\") pod \"f8efc049-5c75-4627-b33c-b0407d1a2067\" (UID: \"f8efc049-5c75-4627-b33c-b0407d1a2067\") " Oct 30 06:59:17.181906 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.181836 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8efc049-5c75-4627-b33c-b0407d1a2067-plugins-config-volume" pod "f8efc049-5c75-4627-b33c-b0407d1a2067" (UID: "f8efc049-5c75-4627-b33c-b0407d1a2067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:17.181994 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.181960 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b9d11a-e96d-4733-b920-2ecf04bee128-plugins-config-volume" pod "f5b9d11a-e96d-4733-b920-2ecf04bee128" (UID: "f5b9d11a-e96d-4733-b920-2ecf04bee128"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:17.182058 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.182041 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8efc049-5c75-4627-b33c-b0407d1a2067-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.182321 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.182284 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03aa02fa-ef29-4efb-ae52-5ed210c4a404-plugins-config-volume" pod "03aa02fa-ef29-4efb-ae52-5ed210c4a404" (UID: "03aa02fa-ef29-4efb-ae52-5ed210c4a404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:17.183715 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.183683 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03aa02fa-ef29-4efb-ae52-5ed210c4a404-kube-api-access-8zvzv" pod "03aa02fa-ef29-4efb-ae52-5ed210c4a404" (UID: "03aa02fa-ef29-4efb-ae52-5ed210c4a404"). InnerVolumeSpecName "kube-api-access-8zvzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:17.184064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.184043 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b9d11a-e96d-4733-b920-2ecf04bee128-kube-api-access-xncj8" pod "f5b9d11a-e96d-4733-b920-2ecf04bee128" (UID: "f5b9d11a-e96d-4733-b920-2ecf04bee128"). InnerVolumeSpecName "kube-api-access-xncj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:17.184064 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.184054 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8efc049-5c75-4627-b33c-b0407d1a2067-kube-api-access-2wvzr" pod "f8efc049-5c75-4627-b33c-b0407d1a2067" (UID: "f8efc049-5c75-4627-b33c-b0407d1a2067"). InnerVolumeSpecName "kube-api-access-2wvzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:17.283375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.283290 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zvzv\" (UniqueName: \"kubernetes.io/projected/03aa02fa-ef29-4efb-ae52-5ed210c4a404-kube-api-access-8zvzv\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.283375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.283339 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b9d11a-e96d-4733-b920-2ecf04bee128-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.283375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.283349 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03aa02fa-ef29-4efb-ae52-5ed210c4a404-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.283375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.283359 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wvzr\" (UniqueName: \"kubernetes.io/projected/f8efc049-5c75-4627-b33c-b0407d1a2067-kube-api-access-2wvzr\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.283375 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.283370 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xncj8\" (UniqueName: \"kubernetes.io/projected/f5b9d11a-e96d-4733-b920-2ecf04bee128-kube-api-access-xncj8\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.620324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.620236 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:59:17.620324 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.620293 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:59:17.874425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.874345 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" Oct 30 06:59:17.874425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.874356 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl" event={"ID":"f8efc049-5c75-4627-b33c-b0407d1a2067","Type":"ContainerDied","Data":"cfb3bfbe9120642e07e199d379ed26d099c2cf21041027ad30a2022a7334e6ac"} Oct 30 06:59:17.874425 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.874418 4431 scope.go:122] "RemoveContainer" containerID="e07b7acb537e23ce9d5a725f52d573448803ecceac78666421af141584755bd3" Oct 30 06:59:17.875917 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.875888 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" event={"ID":"f5b9d11a-e96d-4733-b920-2ecf04bee128","Type":"ContainerDied","Data":"35e30903bf0ec0db5a4400055685331f6d557f3965e7aaf3d280af5dea5fc2c4"} Oct 30 06:59:17.876011 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.875899 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt" Oct 30 06:59:17.877546 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.877525 4431 generic.go:356] "Generic (PLEG): container finished" podID="03aa02fa-ef29-4efb-ae52-5ed210c4a404" containerID="cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80" exitCode=0 Oct 30 06:59:17.877653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.877578 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" Oct 30 06:59:17.877653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.877592 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" event={"ID":"03aa02fa-ef29-4efb-ae52-5ed210c4a404","Type":"ContainerDied","Data":"cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80"} Oct 30 06:59:17.877653 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.877626 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr" event={"ID":"03aa02fa-ef29-4efb-ae52-5ed210c4a404","Type":"ContainerDied","Data":"1d02da6a3898838ee079f24f3b89ba137753838b346bb0e5818937736ba6f197"} Oct 30 06:59:17.884512 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.884481 4431 scope.go:122] "RemoveContainer" containerID="48bbca1a8d749e10307bbbc3858d2c478da536b89c4584326d7a7674f3c103e3" Oct 30 06:59:17.892690 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.892662 4431 scope.go:122] "RemoveContainer" containerID="cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80" Oct 30 06:59:17.901809 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.901789 4431 scope.go:122] "RemoveContainer" containerID="cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80" Oct 30 06:59:17.902052 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:59:17.902033 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80\": container with ID starting with cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80 not found: ID does not exist" containerID="cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80" Oct 30 06:59:17.902105 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.902060 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80"} err="failed to get container status \"cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80\": rpc error: code = NotFound desc = could not find container \"cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80\": container with ID starting with cf479c297ae9ccacf86672c99c697add6191a51cfb500d7e3f934973f6179d80 not found: ID does not exist" Oct 30 06:59:17.926009 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.925982 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl"] Oct 30 06:59:17.937220 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.937196 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69f3fa-f7fbffb7-8cxxl"] Oct 30 06:59:17.967073 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.967048 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt"] Oct 30 06:59:17.970734 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:17.970708 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69f3fa-584957b64d-7cdxt"] Oct 30 06:59:18.004421 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.004399 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr"] Oct 30 06:59:18.008691 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.008667 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6d1778-788b49b675-84mwr"] Oct 30 06:59:18.385887 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.385862 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:59:18.495189 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495162 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-envoy\") pod \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " Oct 30 06:59:18.495370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495217 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istiod-ca-cert\") pod \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " Oct 30 06:59:18.495370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495252 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-podinfo\") pod \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " Oct 30 06:59:18.495370 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495363 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-socket\") pod \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " Oct 30 06:59:18.495579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495407 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-kube-api-access-5lwnr\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-kube-api-access-5lwnr\") pod \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " Oct 30 06:59:18.495579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495437 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-certs\") pod \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " Oct 30 06:59:18.495579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495466 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-token\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-token\") pod \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " Oct 30 06:59:18.495579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495500 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-credential-socket\") pod \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " Oct 30 06:59:18.495579 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495535 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-data\") pod \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\" (UID: \"253e2111-a9ac-4ddf-a3f8-6e86965c99aa\") " Oct 30 06:59:18.495858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495727 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-socket" pod "253e2111-a9ac-4ddf-a3f8-6e86965c99aa" (UID: "253e2111-a9ac-4ddf-a3f8-6e86965c99aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:18.495858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495738 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istiod-ca-cert" pod "253e2111-a9ac-4ddf-a3f8-6e86965c99aa" (UID: "253e2111-a9ac-4ddf-a3f8-6e86965c99aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:18.495858 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495744 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-certs" pod "253e2111-a9ac-4ddf-a3f8-6e86965c99aa" (UID: "253e2111-a9ac-4ddf-a3f8-6e86965c99aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:18.496018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495942 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-credential-socket" pod "253e2111-a9ac-4ddf-a3f8-6e86965c99aa" (UID: "253e2111-a9ac-4ddf-a3f8-6e86965c99aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:18.496018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495983 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-data" pod "253e2111-a9ac-4ddf-a3f8-6e86965c99aa" (UID: "253e2111-a9ac-4ddf-a3f8-6e86965c99aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:18.496018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.495991 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.496018 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.496007 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.496194 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.496019 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.497993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.497967 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-kube-api-access-5lwnr" pod "253e2111-a9ac-4ddf-a3f8-6e86965c99aa" (UID: "253e2111-a9ac-4ddf-a3f8-6e86965c99aa"). InnerVolumeSpecName "kube-api-access-5lwnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:18.497993 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.497962 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-podinfo" pod "253e2111-a9ac-4ddf-a3f8-6e86965c99aa" (UID: "253e2111-a9ac-4ddf-a3f8-6e86965c99aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:59:18.498130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.498024 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-envoy" pod "253e2111-a9ac-4ddf-a3f8-6e86965c99aa" (UID: "253e2111-a9ac-4ddf-a3f8-6e86965c99aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:18.498130 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.498035 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-token" pod "253e2111-a9ac-4ddf-a3f8-6e86965c99aa" (UID: "253e2111-a9ac-4ddf-a3f8-6e86965c99aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:18.597359 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.597336 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lwnr\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-kube-api-access-5lwnr\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.597359 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.597356 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.597510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.597367 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.597510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.597376 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.597510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.597386 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.597510 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.597393 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253e2111-a9ac-4ddf-a3f8-6e86965c99aa-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.886396 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.886323 4431 generic.go:356] "Generic (PLEG): container finished" podID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerID="ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee" exitCode=0 Oct 30 06:59:18.886396 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.886383 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" Oct 30 06:59:18.886824 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.886399 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" event={"ID":"253e2111-a9ac-4ddf-a3f8-6e86965c99aa","Type":"ContainerDied","Data":"ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee"} Oct 30 06:59:18.886824 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.886438 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz" event={"ID":"253e2111-a9ac-4ddf-a3f8-6e86965c99aa","Type":"ContainerDied","Data":"377577f7ea6ae3cbb546728b68254202d09459d95f216efd497d7fb5b0d4b1b0"} Oct 30 06:59:18.886824 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.886462 4431 scope.go:122] "RemoveContainer" containerID="ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee" Oct 30 06:59:18.896281 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.896266 4431 scope.go:122] "RemoveContainer" containerID="abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4" Oct 30 06:59:18.904819 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.904799 4431 scope.go:122] "RemoveContainer" containerID="ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee" Oct 30 06:59:18.905092 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:59:18.905063 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee\": container with ID starting with ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee not found: ID does not exist" containerID="ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee" Oct 30 06:59:18.905146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.905101 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee"} err="failed to get container status \"ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee\": rpc error: code = NotFound desc = could not find container \"ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee\": container with ID starting with ab2e82bb7380e0e057ca087115e6585d579fbbf744b3e314a2d15450b46b67ee not found: ID does not exist" Oct 30 06:59:18.905146 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.905115 4431 scope.go:122] "RemoveContainer" containerID="abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4" Oct 30 06:59:18.905381 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:59:18.905365 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4\": container with ID starting with abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4 not found: ID does not exist" containerID="abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4" Oct 30 06:59:18.905443 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.905384 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4"} err="failed to get container status \"abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4\": rpc error: code = NotFound desc = could not find container \"abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4\": container with ID starting with abbcba261efdd3f68c39be6b8af7c79466873b83d1303fb2b7db28487e54cfc4 not found: ID does not exist" Oct 30 06:59:18.937902 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.937878 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz"] Oct 30 06:59:18.949574 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:18.949552 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6d1778-openshift-default-5fccc76b5c-ztjmz"] Oct 30 06:59:19.336964 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:19.336935 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03aa02fa-ef29-4efb-ae52-5ed210c4a404" path="/var/lib/kubelet/pods/03aa02fa-ef29-4efb-ae52-5ed210c4a404/volumes" Oct 30 06:59:19.337419 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:19.337404 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" path="/var/lib/kubelet/pods/253e2111-a9ac-4ddf-a3f8-6e86965c99aa/volumes" Oct 30 06:59:19.337767 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:19.337754 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b9d11a-e96d-4733-b920-2ecf04bee128" path="/var/lib/kubelet/pods/f5b9d11a-e96d-4733-b920-2ecf04bee128/volumes" Oct 30 06:59:19.338426 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:19.338409 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8efc049-5c75-4627-b33c-b0407d1a2067" path="/var/lib/kubelet/pods/f8efc049-5c75-4627-b33c-b0407d1a2067/volumes" Oct 30 06:59:33.701110 ip-10-0-136-123 kubenswrapper[4431]: E1030 06:59:33.701071 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 06:59:57.795137 ip-10-0-136-123 kubenswrapper[4431]: I1030 06:59:57.795056 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7"] Oct 30 07:00:01.984892 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:01.984852 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz"] Oct 30 07:00:02.795586 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:02.795537 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" podUID="4b4385bc-6024-4651-8422-f785697b8049" containerName="payload-processing" containerID="cri-o://ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89" gracePeriod=25 Oct 30 07:00:03.702220 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:03.702190 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 07:00:06.985254 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:06.985197 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" podUID="62504278-ebeb-48fd-a9a3-6a68dff619e4" containerName="payload-processing" containerID="cri-o://80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142" gracePeriod=25 Oct 30 07:00:09.842502 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:09.842461 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" podUID="4b4385bc-6024-4651-8422-f785697b8049" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:13.025725 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:13.025645 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8"] Oct 30 07:00:16.446285 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:16.446245 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6"] Oct 30 07:00:16.458238 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:16.458208 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn"] Oct 30 07:00:16.754708 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:16.754618 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" podUID="62504278-ebeb-48fd-a9a3-6a68dff619e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:18.822831 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:18.822797 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9"] Oct 30 07:00:19.842821 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:19.842785 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" podUID="4b4385bc-6024-4651-8422-f785697b8049" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:22.221007 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:22.220971 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt"] Oct 30 07:00:23.063346 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.063274 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 07:00:23.157658 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.157625 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4385bc-6024-4651-8422-f785697b8049-kube-api-access-8pr7g\" (UniqueName: \"kubernetes.io/projected/4b4385bc-6024-4651-8422-f785697b8049-kube-api-access-8pr7g\") pod \"4b4385bc-6024-4651-8422-f785697b8049\" (UID: \"4b4385bc-6024-4651-8422-f785697b8049\") " Oct 30 07:00:23.157806 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.157710 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4385bc-6024-4651-8422-f785697b8049-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4385bc-6024-4651-8422-f785697b8049-plugins-config-volume\") pod \"4b4385bc-6024-4651-8422-f785697b8049\" (UID: \"4b4385bc-6024-4651-8422-f785697b8049\") " Oct 30 07:00:23.158044 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.158017 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4385bc-6024-4651-8422-f785697b8049-plugins-config-volume" pod "4b4385bc-6024-4651-8422-f785697b8049" (UID: "4b4385bc-6024-4651-8422-f785697b8049"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:23.159751 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.159729 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4385bc-6024-4651-8422-f785697b8049-kube-api-access-8pr7g" pod "4b4385bc-6024-4651-8422-f785697b8049" (UID: "4b4385bc-6024-4651-8422-f785697b8049"). InnerVolumeSpecName "kube-api-access-8pr7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:23.228067 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.228036 4431 generic.go:356] "Generic (PLEG): container finished" podID="4b4385bc-6024-4651-8422-f785697b8049" containerID="ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89" exitCode=0 Oct 30 07:00:23.228428 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.228088 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" Oct 30 07:00:23.228428 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.228086 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" event={"ID":"4b4385bc-6024-4651-8422-f785697b8049","Type":"ContainerDied","Data":"ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89"} Oct 30 07:00:23.228428 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.228251 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7" event={"ID":"4b4385bc-6024-4651-8422-f785697b8049","Type":"ContainerDied","Data":"ab227889f163f6fd6103a2dc7de21cf3b86851c82a35c7438626eccb0fca91b4"} Oct 30 07:00:23.228428 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.228290 4431 scope.go:122] "RemoveContainer" containerID="ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89" Oct 30 07:00:23.237501 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.237487 4431 scope.go:122] "RemoveContainer" containerID="ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89" Oct 30 07:00:23.237759 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:23.237743 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89\": container with ID starting with ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89 not found: ID does not exist" containerID="ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89" Oct 30 07:00:23.237839 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.237767 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89"} err="failed to get container status \"ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89\": rpc error: code = NotFound desc = could not find container \"ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89\": container with ID starting with ea56191775801cfe7022b4cf98c4b351f5b8ca43bac433a20106329b882cfa89 not found: ID does not exist" Oct 30 07:00:23.258880 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.258856 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pr7g\" (UniqueName: \"kubernetes.io/projected/4b4385bc-6024-4651-8422-f785697b8049-kube-api-access-8pr7g\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:23.258880 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.258880 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4385bc-6024-4651-8422-f785697b8049-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:23.284880 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.284848 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7"] Oct 30 07:00:23.291280 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.291254 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-868b1f-fcffcf6bd-sp6f7"] Oct 30 07:00:23.337225 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.337157 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4385bc-6024-4651-8422-f785697b8049" path="/var/lib/kubelet/pods/4b4385bc-6024-4651-8422-f785697b8049/volumes" Oct 30 07:00:23.824089 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:23.824059 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" podUID="0cb21f6c-d934-451c-ae6e-e878af83cc49" containerName="payload-processing" containerID="cri-o://1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9" gracePeriod=25 Oct 30 07:00:26.178381 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:26.178347 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx"] Oct 30 07:00:26.178844 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:26.178565 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" containerID="cri-o://ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902" gracePeriod=30 Oct 30 07:00:26.718591 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:26.718557 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:00:26.718770 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:26.718627 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:00:26.755120 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:26.755084 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" podUID="62504278-ebeb-48fd-a9a3-6a68dff619e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:27.222030 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.221995 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" podUID="3c5462b6-b727-4176-af24-18f12c02d3c2" containerName="payload-processing" containerID="cri-o://3d731db4a89fa9b9efe99eb7f730a10faec5d68ad63f810a680c255e0038f97d" gracePeriod=25 Oct 30 07:00:27.235239 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.235217 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 07:00:27.248459 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.248413 4431 generic.go:356] "Generic (PLEG): container finished" podID="62504278-ebeb-48fd-a9a3-6a68dff619e4" containerID="80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142" exitCode=0 Oct 30 07:00:27.248459 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.248451 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" Oct 30 07:00:27.248595 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.248497 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" event={"ID":"62504278-ebeb-48fd-a9a3-6a68dff619e4","Type":"ContainerDied","Data":"80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142"} Oct 30 07:00:27.248595 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.248580 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz" event={"ID":"62504278-ebeb-48fd-a9a3-6a68dff619e4","Type":"ContainerDied","Data":"90282d61542e2774f90aea17b84b7e196c421caaff20eaa5777b08a59944e614"} Oct 30 07:00:27.248663 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.248606 4431 scope.go:122] "RemoveContainer" containerID="80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142" Oct 30 07:00:27.258209 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.258191 4431 scope.go:122] "RemoveContainer" containerID="80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142" Oct 30 07:00:27.258590 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:27.258571 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142\": container with ID starting with 80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142 not found: ID does not exist" containerID="80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142" Oct 30 07:00:27.258663 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.258603 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142"} err="failed to get container status \"80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142\": rpc error: code = NotFound desc = could not find container \"80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142\": container with ID starting with 80f285fb629563e9756f161196f3ab7488c8f499feaee5cf90e1f45ba4031142 not found: ID does not exist" Oct 30 07:00:27.306654 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.306629 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62504278-ebeb-48fd-a9a3-6a68dff619e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62504278-ebeb-48fd-a9a3-6a68dff619e4-plugins-config-volume\") pod \"62504278-ebeb-48fd-a9a3-6a68dff619e4\" (UID: \"62504278-ebeb-48fd-a9a3-6a68dff619e4\") " Oct 30 07:00:27.306839 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.306821 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62504278-ebeb-48fd-a9a3-6a68dff619e4-kube-api-access-cb2r5\" (UniqueName: \"kubernetes.io/projected/62504278-ebeb-48fd-a9a3-6a68dff619e4-kube-api-access-cb2r5\") pod \"62504278-ebeb-48fd-a9a3-6a68dff619e4\" (UID: \"62504278-ebeb-48fd-a9a3-6a68dff619e4\") " Oct 30 07:00:27.307044 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.307025 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62504278-ebeb-48fd-a9a3-6a68dff619e4-plugins-config-volume" pod "62504278-ebeb-48fd-a9a3-6a68dff619e4" (UID: "62504278-ebeb-48fd-a9a3-6a68dff619e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:27.307182 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.307168 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62504278-ebeb-48fd-a9a3-6a68dff619e4-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:27.308887 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.308860 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62504278-ebeb-48fd-a9a3-6a68dff619e4-kube-api-access-cb2r5" pod "62504278-ebeb-48fd-a9a3-6a68dff619e4" (UID: "62504278-ebeb-48fd-a9a3-6a68dff619e4"). InnerVolumeSpecName "kube-api-access-cb2r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:27.408621 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.408586 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2r5\" (UniqueName: \"kubernetes.io/projected/62504278-ebeb-48fd-a9a3-6a68dff619e4-kube-api-access-cb2r5\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:27.587894 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.587830 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz"] Oct 30 07:00:27.592253 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:27.592216 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-30fbb6-649cfc46f8-7g7mz"] Oct 30 07:00:28.046364 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.046326 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" podUID="0f0c53ac-dd7d-4ee0-b707-51857ccf2724" containerName="main" containerID="cri-o://2a591ad2c50a1e4154ffb4fcd6c747d0b8e70cf1321ab4d81d2692974fa6b5ab" gracePeriod=45 Oct 30 07:00:28.255495 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.255467 4431 generic.go:356] "Generic (PLEG): container finished" podID="0f0c53ac-dd7d-4ee0-b707-51857ccf2724" containerID="2a591ad2c50a1e4154ffb4fcd6c747d0b8e70cf1321ab4d81d2692974fa6b5ab" exitCode=0 Oct 30 07:00:28.255878 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.255552 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" event={"ID":"0f0c53ac-dd7d-4ee0-b707-51857ccf2724","Type":"ContainerDied","Data":"2a591ad2c50a1e4154ffb4fcd6c747d0b8e70cf1321ab4d81d2692974fa6b5ab"} Oct 30 07:00:28.293166 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.293145 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 07:00:28.419986 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.419906 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tmp-dir\") pod \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " Oct 30 07:00:28.419986 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.419973 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-dshm\") pod \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " Oct 30 07:00:28.420226 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420005 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-home\") pod \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " Oct 30 07:00:28.420226 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420144 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-model-cache\") pod \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " Oct 30 07:00:28.420412 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420226 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tls-certs\") pod \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " Oct 30 07:00:28.420412 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420274 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-kube-api-access-ph8xb\" (UniqueName: \"kubernetes.io/projected/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-kube-api-access-ph8xb\") pod \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\" (UID: \"0f0c53ac-dd7d-4ee0-b707-51857ccf2724\") " Oct 30 07:00:28.420533 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420218 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tmp-dir" pod "0f0c53ac-dd7d-4ee0-b707-51857ccf2724" (UID: "0f0c53ac-dd7d-4ee0-b707-51857ccf2724"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.420674 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420430 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-model-cache" pod "0f0c53ac-dd7d-4ee0-b707-51857ccf2724" (UID: "0f0c53ac-dd7d-4ee0-b707-51857ccf2724"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.420674 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420641 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-home" pod "0f0c53ac-dd7d-4ee0-b707-51857ccf2724" (UID: "0f0c53ac-dd7d-4ee0-b707-51857ccf2724"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.421005 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420942 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.421005 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420972 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.421005 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.420986 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.424296 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.424272 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-dshm" pod "0f0c53ac-dd7d-4ee0-b707-51857ccf2724" (UID: "0f0c53ac-dd7d-4ee0-b707-51857ccf2724"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.424420 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.424292 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tls-certs" pod "0f0c53ac-dd7d-4ee0-b707-51857ccf2724" (UID: "0f0c53ac-dd7d-4ee0-b707-51857ccf2724"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:28.424420 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.424296 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-kube-api-access-ph8xb" pod "0f0c53ac-dd7d-4ee0-b707-51857ccf2724" (UID: "0f0c53ac-dd7d-4ee0-b707-51857ccf2724"). InnerVolumeSpecName "kube-api-access-ph8xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:28.522237 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.522205 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.522237 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.522234 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8xb\" (UniqueName: \"kubernetes.io/projected/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-kube-api-access-ph8xb\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.522441 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:28.522244 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c53ac-dd7d-4ee0-b707-51857ccf2724-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:29.263194 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:29.263164 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" Oct 30 07:00:29.263731 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:29.263205 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8" event={"ID":"0f0c53ac-dd7d-4ee0-b707-51857ccf2724","Type":"ContainerDied","Data":"69ed716df3beac64fb6c8ae446e92e7b15579a4575d427d3f716da579fb43892"} Oct 30 07:00:29.263731 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:29.263258 4431 scope.go:122] "RemoveContainer" containerID="2a591ad2c50a1e4154ffb4fcd6c747d0b8e70cf1321ab4d81d2692974fa6b5ab" Oct 30 07:00:29.313993 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:29.313967 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8"] Oct 30 07:00:29.325993 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:29.325968 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/premium-w2-aaf467-kserve-d8d84c77c-v6bn8"] Oct 30 07:00:29.336181 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:29.336159 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0c53ac-dd7d-4ee0-b707-51857ccf2724" path="/var/lib/kubelet/pods/0f0c53ac-dd7d-4ee0-b707-51857ccf2724/volumes" Oct 30 07:00:29.336579 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:29.336564 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62504278-ebeb-48fd-a9a3-6a68dff619e4" path="/var/lib/kubelet/pods/62504278-ebeb-48fd-a9a3-6a68dff619e4/volumes" Oct 30 07:00:29.621146 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:29.621070 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" podUID="0cb21f6c-d934-451c-ae6e-e878af83cc49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:31.425552 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.425525 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 07:00:31.466989 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.466908 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" podUID="d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" containerName="main" containerID="cri-o://c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4" gracePeriod=45 Oct 30 07:00:31.477689 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.477647 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" podUID="80db6c09-1940-4274-a21c-57ab7dd6738c" containerName="main" containerID="cri-o://fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978" gracePeriod=45 Oct 30 07:00:31.550158 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550130 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a20d2777-0ed1-4ead-a144-799c102a6d97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20d2777-0ed1-4ead-a144-799c102a6d97-istiod-ca-cert\") pod \"a20d2777-0ed1-4ead-a144-799c102a6d97\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " Oct 30 07:00:31.550274 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550165 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-certs\") pod \"a20d2777-0ed1-4ead-a144-799c102a6d97\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " Oct 30 07:00:31.550274 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550193 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-podinfo\") pod \"a20d2777-0ed1-4ead-a144-799c102a6d97\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " Oct 30 07:00:31.550274 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550266 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-socket\") pod \"a20d2777-0ed1-4ead-a144-799c102a6d97\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " Oct 30 07:00:31.550544 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550428 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-credential-socket\") pod \"a20d2777-0ed1-4ead-a144-799c102a6d97\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " Oct 30 07:00:31.550544 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550441 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-certs" pod "a20d2777-0ed1-4ead-a144-799c102a6d97" (UID: "a20d2777-0ed1-4ead-a144-799c102a6d97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.550544 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550477 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-envoy\") pod \"a20d2777-0ed1-4ead-a144-799c102a6d97\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " Oct 30 07:00:31.550544 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550520 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-token\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-token\") pod \"a20d2777-0ed1-4ead-a144-799c102a6d97\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " Oct 30 07:00:31.550777 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550598 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-kube-api-access-prvl9\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-kube-api-access-prvl9\") pod \"a20d2777-0ed1-4ead-a144-799c102a6d97\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " Oct 30 07:00:31.550777 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550601 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-socket" pod "a20d2777-0ed1-4ead-a144-799c102a6d97" (UID: "a20d2777-0ed1-4ead-a144-799c102a6d97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.550777 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550680 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-data\") pod \"a20d2777-0ed1-4ead-a144-799c102a6d97\" (UID: \"a20d2777-0ed1-4ead-a144-799c102a6d97\") " Oct 30 07:00:31.550777 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550766 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-credential-socket" pod "a20d2777-0ed1-4ead-a144-799c102a6d97" (UID: "a20d2777-0ed1-4ead-a144-799c102a6d97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.550997 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.550663 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20d2777-0ed1-4ead-a144-799c102a6d97-istiod-ca-cert" pod "a20d2777-0ed1-4ead-a144-799c102a6d97" (UID: "a20d2777-0ed1-4ead-a144-799c102a6d97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:31.551070 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.551049 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.551129 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.551072 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.551129 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.551089 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a20d2777-0ed1-4ead-a144-799c102a6d97-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.551129 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.551093 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-data" pod "a20d2777-0ed1-4ead-a144-799c102a6d97" (UID: "a20d2777-0ed1-4ead-a144-799c102a6d97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.551129 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.551103 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.552619 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.552598 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-kube-api-access-prvl9" pod "a20d2777-0ed1-4ead-a144-799c102a6d97" (UID: "a20d2777-0ed1-4ead-a144-799c102a6d97"). InnerVolumeSpecName "kube-api-access-prvl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:31.552706 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.552617 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-token" pod "a20d2777-0ed1-4ead-a144-799c102a6d97" (UID: "a20d2777-0ed1-4ead-a144-799c102a6d97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:31.552861 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.552845 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-envoy" pod "a20d2777-0ed1-4ead-a144-799c102a6d97" (UID: "a20d2777-0ed1-4ead-a144-799c102a6d97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.552907 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.552856 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-podinfo" pod "a20d2777-0ed1-4ead-a144-799c102a6d97" (UID: "a20d2777-0ed1-4ead-a144-799c102a6d97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:00:31.652500 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.652467 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.652500 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.652499 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.652676 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.652509 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.652676 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.652517 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.652676 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.652526 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvl9\" (UniqueName: \"kubernetes.io/projected/a20d2777-0ed1-4ead-a144-799c102a6d97-kube-api-access-prvl9\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.685131 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.685102 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f"] Oct 30 07:00:31.706728 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.706698 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524"] Oct 30 07:00:31.719253 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.719200 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 07:00:31.748767 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.748746 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 07:00:31.854959 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.854854 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tmp-dir\") pod \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " Oct 30 07:00:31.855443 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.854999 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80db6c09-1940-4274-a21c-57ab7dd6738c-kube-api-access-fqmpq\" (UniqueName: \"kubernetes.io/projected/80db6c09-1940-4274-a21c-57ab7dd6738c-kube-api-access-fqmpq\") pod \"80db6c09-1940-4274-a21c-57ab7dd6738c\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " Oct 30 07:00:31.855443 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855048 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-home\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-home\") pod \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " Oct 30 07:00:31.855443 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855153 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-dshm\") pod \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " Oct 30 07:00:31.855443 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855331 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-tmp-dir\") pod \"80db6c09-1940-4274-a21c-57ab7dd6738c\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " Oct 30 07:00:31.855443 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855419 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-dshm\") pod \"80db6c09-1940-4274-a21c-57ab7dd6738c\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " Oct 30 07:00:31.855711 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855450 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-kube-api-access-25l6h\" (UniqueName: \"kubernetes.io/projected/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-kube-api-access-25l6h\") pod \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " Oct 30 07:00:31.855711 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855493 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80db6c09-1940-4274-a21c-57ab7dd6738c-tls-certs\" (UniqueName: \"kubernetes.io/secret/80db6c09-1940-4274-a21c-57ab7dd6738c-tls-certs\") pod \"80db6c09-1940-4274-a21c-57ab7dd6738c\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " Oct 30 07:00:31.855711 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855537 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tls-certs\") pod \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " Oct 30 07:00:31.855711 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855167 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tmp-dir" pod "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" (UID: "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.855711 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855585 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-home\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-home\") pod \"80db6c09-1940-4274-a21c-57ab7dd6738c\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " Oct 30 07:00:31.855711 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855638 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-model-cache\") pod \"80db6c09-1940-4274-a21c-57ab7dd6738c\" (UID: \"80db6c09-1940-4274-a21c-57ab7dd6738c\") " Oct 30 07:00:31.855936 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.855714 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-model-cache\") pod \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\" (UID: \"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf\") " Oct 30 07:00:31.856144 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.856116 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.856271 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.856249 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-home" pod "80db6c09-1940-4274-a21c-57ab7dd6738c" (UID: "80db6c09-1940-4274-a21c-57ab7dd6738c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.856671 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.856416 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-model-cache" pod "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" (UID: "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.856790 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.856743 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-model-cache" pod "80db6c09-1940-4274-a21c-57ab7dd6738c" (UID: "80db6c09-1940-4274-a21c-57ab7dd6738c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.858254 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.858220 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80db6c09-1940-4274-a21c-57ab7dd6738c-kube-api-access-fqmpq" pod "80db6c09-1940-4274-a21c-57ab7dd6738c" (UID: "80db6c09-1940-4274-a21c-57ab7dd6738c"). InnerVolumeSpecName "kube-api-access-fqmpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:31.858866 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.858832 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tls-certs" pod "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" (UID: "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:31.859463 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.859434 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-tmp-dir" pod "80db6c09-1940-4274-a21c-57ab7dd6738c" (UID: "80db6c09-1940-4274-a21c-57ab7dd6738c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.859570 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.859457 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-dshm" pod "80db6c09-1940-4274-a21c-57ab7dd6738c" (UID: "80db6c09-1940-4274-a21c-57ab7dd6738c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.859820 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.859798 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-home" pod "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" (UID: "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.860142 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.860116 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-dshm" pod "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" (UID: "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.860259 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.860219 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-kube-api-access-25l6h" pod "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" (UID: "d64c38d6-979d-4e74-b2ea-ba1d34ccecaf"). InnerVolumeSpecName "kube-api-access-25l6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:31.861623 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.861598 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80db6c09-1940-4274-a21c-57ab7dd6738c-tls-certs" pod "80db6c09-1940-4274-a21c-57ab7dd6738c" (UID: "80db6c09-1940-4274-a21c-57ab7dd6738c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:31.957865 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957838 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.957865 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957863 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.958051 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957873 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.958051 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957883 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqmpq\" (UniqueName: \"kubernetes.io/projected/80db6c09-1940-4274-a21c-57ab7dd6738c-kube-api-access-fqmpq\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.958051 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957892 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.958051 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957899 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.958051 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957906 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.958051 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957914 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80db6c09-1940-4274-a21c-57ab7dd6738c-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.958051 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957921 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25l6h\" (UniqueName: \"kubernetes.io/projected/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-kube-api-access-25l6h\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.958051 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957928 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80db6c09-1940-4274-a21c-57ab7dd6738c-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.958051 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:31.957937 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:32.280146 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.280119 4431 generic.go:356] "Generic (PLEG): container finished" podID="d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" containerID="c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4" exitCode=0 Oct 30 07:00:32.280326 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.280168 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" Oct 30 07:00:32.280326 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.280198 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" event={"ID":"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf","Type":"ContainerDied","Data":"c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4"} Oct 30 07:00:32.280326 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.280237 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6" event={"ID":"d64c38d6-979d-4e74-b2ea-ba1d34ccecaf","Type":"ContainerDied","Data":"d86c5adc5178363eaed917ea26701cdbfd806f03c3d6f8ff6715f1dad2a3eff1"} Oct 30 07:00:32.280326 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.280258 4431 scope.go:122] "RemoveContainer" containerID="c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4" Oct 30 07:00:32.281744 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.281719 4431 generic.go:356] "Generic (PLEG): container finished" podID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerID="ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902" exitCode=0 Oct 30 07:00:32.281868 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.281757 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" Oct 30 07:00:32.281868 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.281782 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" event={"ID":"a20d2777-0ed1-4ead-a144-799c102a6d97","Type":"ContainerDied","Data":"ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902"} Oct 30 07:00:32.281868 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.281830 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx" event={"ID":"a20d2777-0ed1-4ead-a144-799c102a6d97","Type":"ContainerDied","Data":"889ae609853b76515c4fc52a3f91350999b130315ebabaf4576c32f5d4bcc08f"} Oct 30 07:00:32.289768 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.289747 4431 generic.go:356] "Generic (PLEG): container finished" podID="80db6c09-1940-4274-a21c-57ab7dd6738c" containerID="fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978" exitCode=0 Oct 30 07:00:32.289879 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.289793 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" event={"ID":"80db6c09-1940-4274-a21c-57ab7dd6738c","Type":"ContainerDied","Data":"fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978"} Oct 30 07:00:32.289879 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.289802 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" Oct 30 07:00:32.289879 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.289823 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn" event={"ID":"80db6c09-1940-4274-a21c-57ab7dd6738c","Type":"ContainerDied","Data":"9260af1c6a23aeae3e38fc40f7fcd71f2aa51b76f9d345f6c7592e31ea0702db"} Oct 30 07:00:32.290480 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.290463 4431 scope.go:122] "RemoveContainer" containerID="c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4" Oct 30 07:00:32.290801 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:32.290733 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4\": container with ID starting with c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4 not found: ID does not exist" containerID="c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4" Oct 30 07:00:32.290801 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.290763 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4"} err="failed to get container status \"c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4\": rpc error: code = NotFound desc = could not find container \"c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4\": container with ID starting with c44239d7c69daf604b134a158e853a65929f478bb509af63e4984b33e579dac4 not found: ID does not exist" Oct 30 07:00:32.290801 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.290786 4431 scope.go:122] "RemoveContainer" containerID="ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902" Oct 30 07:00:32.301041 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.300837 4431 scope.go:122] "RemoveContainer" containerID="ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902" Oct 30 07:00:32.301124 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:32.301106 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902\": container with ID starting with ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902 not found: ID does not exist" containerID="ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902" Oct 30 07:00:32.301166 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.301130 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902"} err="failed to get container status \"ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902\": rpc error: code = NotFound desc = could not find container \"ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902\": container with ID starting with ddba4054b3bdfeaf8983b994b8d2b29a66e92409d23b6fbcf5d91a5ec8cb3902 not found: ID does not exist" Oct 30 07:00:32.301166 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.301147 4431 scope.go:122] "RemoveContainer" containerID="fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978" Oct 30 07:00:32.308361 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.308346 4431 scope.go:122] "RemoveContainer" containerID="fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978" Oct 30 07:00:32.308620 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:32.308603 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978\": container with ID starting with fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978 not found: ID does not exist" containerID="fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978" Oct 30 07:00:32.308667 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.308626 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978"} err="failed to get container status \"fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978\": rpc error: code = NotFound desc = could not find container \"fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978\": container with ID starting with fb7e904f622ecbc17830a45421f1ca7da8c902b70ece475dedf8ab5233bed978 not found: ID does not exist" Oct 30 07:00:32.330784 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.330757 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx"] Oct 30 07:00:32.342437 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.342417 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fa5a69-openshift-default-6548c4d875-rlppx"] Oct 30 07:00:32.376005 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.375981 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6"] Oct 30 07:00:32.380140 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.380119 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/premium-w3-8d6349-kserve-6bb7bd68c-xhlb6"] Oct 30 07:00:32.427161 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.427135 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn"] Oct 30 07:00:32.432895 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:32.432875 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/simulator-w3-8d6349-kserve-75b566966b-fpfzn"] Oct 30 07:00:33.336052 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:33.336024 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80db6c09-1940-4274-a21c-57ab7dd6738c" path="/var/lib/kubelet/pods/80db6c09-1940-4274-a21c-57ab7dd6738c/volumes" Oct 30 07:00:33.336491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:33.336477 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" path="/var/lib/kubelet/pods/a20d2777-0ed1-4ead-a144-799c102a6d97/volumes" Oct 30 07:00:33.336855 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:33.336843 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" path="/var/lib/kubelet/pods/d64c38d6-979d-4e74-b2ea-ba1d34ccecaf/volumes" Oct 30 07:00:33.700068 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:33.700032 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 07:00:36.599443 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:36.599400 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" podUID="3c5462b6-b727-4176-af24-18f12c02d3c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:37.907157 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:37.907122 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs"] Oct 30 07:00:37.907858 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:37.907399 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" podUID="b81c0402-ae16-458d-bdae-659aa77365d9" containerName="maas-api" containerID="cri-o://78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416" gracePeriod=30 Oct 30 07:00:38.151078 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.151051 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 07:00:38.318607 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.318570 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls\") pod \"b81c0402-ae16-458d-bdae-659aa77365d9\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " Oct 30 07:00:38.318784 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.318624 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b81c0402-ae16-458d-bdae-659aa77365d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0402-ae16-458d-bdae-659aa77365d9-openshift-service-ca\") pod \"b81c0402-ae16-458d-bdae-659aa77365d9\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " Oct 30 07:00:38.318784 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.318703 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81c0402-ae16-458d-bdae-659aa77365d9-kube-api-access-lpzrc\" (UniqueName: \"kubernetes.io/projected/b81c0402-ae16-458d-bdae-659aa77365d9-kube-api-access-lpzrc\") pod \"b81c0402-ae16-458d-bdae-659aa77365d9\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " Oct 30 07:00:38.318887 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.318781 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-metrics-certs\") pod \"b81c0402-ae16-458d-bdae-659aa77365d9\" (UID: \"b81c0402-ae16-458d-bdae-659aa77365d9\") " Oct 30 07:00:38.319118 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.319078 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81c0402-ae16-458d-bdae-659aa77365d9-openshift-service-ca" pod "b81c0402-ae16-458d-bdae-659aa77365d9" (UID: "b81c0402-ae16-458d-bdae-659aa77365d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:38.321329 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.321277 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls" pod "b81c0402-ae16-458d-bdae-659aa77365d9" (UID: "b81c0402-ae16-458d-bdae-659aa77365d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:38.321453 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.321378 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81c0402-ae16-458d-bdae-659aa77365d9-kube-api-access-lpzrc" pod "b81c0402-ae16-458d-bdae-659aa77365d9" (UID: "b81c0402-ae16-458d-bdae-659aa77365d9"). InnerVolumeSpecName "kube-api-access-lpzrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:38.322703 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.322671 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-metrics-certs" pod "b81c0402-ae16-458d-bdae-659aa77365d9" (UID: "b81c0402-ae16-458d-bdae-659aa77365d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:38.327621 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.327600 4431 generic.go:356] "Generic (PLEG): container finished" podID="b81c0402-ae16-458d-bdae-659aa77365d9" containerID="78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416" exitCode=0 Oct 30 07:00:38.327726 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.327664 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" Oct 30 07:00:38.327726 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.327691 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" event={"ID":"b81c0402-ae16-458d-bdae-659aa77365d9","Type":"ContainerDied","Data":"78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416"} Oct 30 07:00:38.327799 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.327763 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs" event={"ID":"b81c0402-ae16-458d-bdae-659aa77365d9","Type":"ContainerDied","Data":"cfe60325ce9348c2e13dfe727a7a6b440fad1fd04fc17a9d168fac354f7672dc"} Oct 30 07:00:38.327853 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.327805 4431 scope.go:122] "RemoveContainer" containerID="78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416" Oct 30 07:00:38.344984 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.344967 4431 scope.go:122] "RemoveContainer" containerID="78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416" Oct 30 07:00:38.345265 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:38.345244 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416\": container with ID starting with 78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416 not found: ID does not exist" containerID="78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416" Oct 30 07:00:38.345339 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.345273 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416"} err="failed to get container status \"78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416\": rpc error: code = NotFound desc = could not find container \"78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416\": container with ID starting with 78991f303db6efa1ad209577474930c1b457242a7ac6aac56b334b83dfee0416 not found: ID does not exist" Oct 30 07:00:38.378335 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.378297 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs"] Oct 30 07:00:38.384509 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.384486 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8d6349-d5cfc5d55-tmgcs"] Oct 30 07:00:38.420077 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.420054 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpzrc\" (UniqueName: \"kubernetes.io/projected/b81c0402-ae16-458d-bdae-659aa77365d9-kube-api-access-lpzrc\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:38.420077 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.420076 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:38.420222 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.420086 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81c0402-ae16-458d-bdae-659aa77365d9-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:38.420222 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:38.420095 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0402-ae16-458d-bdae-659aa77365d9-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:39.336679 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:39.336646 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b81c0402-ae16-458d-bdae-659aa77365d9" path="/var/lib/kubelet/pods/b81c0402-ae16-458d-bdae-659aa77365d9/volumes" Oct 30 07:00:39.621016 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:39.620924 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" podUID="0cb21f6c-d934-451c-ae6e-e878af83cc49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:39.960999 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:39.960972 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt"] Oct 30 07:00:39.961253 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:39.961221 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" podUID="13d58406-ef62-4c3e-a222-91f79c761619" containerName="maas-api" containerID="cri-o://ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf" gracePeriod=30 Oct 30 07:00:40.218985 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.218923 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 07:00:40.237391 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.237364 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d58406-ef62-4c3e-a222-91f79c761619-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d58406-ef62-4c3e-a222-91f79c761619-openshift-service-ca\") pod \"13d58406-ef62-4c3e-a222-91f79c761619\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " Oct 30 07:00:40.237521 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.237403 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-metrics-certs\") pod \"13d58406-ef62-4c3e-a222-91f79c761619\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " Oct 30 07:00:40.237521 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.237470 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d58406-ef62-4c3e-a222-91f79c761619-kube-api-access-jt527\" (UniqueName: \"kubernetes.io/projected/13d58406-ef62-4c3e-a222-91f79c761619-kube-api-access-jt527\") pod \"13d58406-ef62-4c3e-a222-91f79c761619\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " Oct 30 07:00:40.237521 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.237504 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-maas-api-tls\") pod \"13d58406-ef62-4c3e-a222-91f79c761619\" (UID: \"13d58406-ef62-4c3e-a222-91f79c761619\") " Oct 30 07:00:40.237716 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.237698 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d58406-ef62-4c3e-a222-91f79c761619-openshift-service-ca" pod "13d58406-ef62-4c3e-a222-91f79c761619" (UID: "13d58406-ef62-4c3e-a222-91f79c761619"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:40.237872 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.237851 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d58406-ef62-4c3e-a222-91f79c761619-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:40.239545 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.239520 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-maas-api-tls" pod "13d58406-ef62-4c3e-a222-91f79c761619" (UID: "13d58406-ef62-4c3e-a222-91f79c761619"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:40.239665 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.239603 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-metrics-certs" pod "13d58406-ef62-4c3e-a222-91f79c761619" (UID: "13d58406-ef62-4c3e-a222-91f79c761619"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:40.239712 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.239669 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d58406-ef62-4c3e-a222-91f79c761619-kube-api-access-jt527" pod "13d58406-ef62-4c3e-a222-91f79c761619" (UID: "13d58406-ef62-4c3e-a222-91f79c761619"). InnerVolumeSpecName "kube-api-access-jt527". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:40.342735 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.339644 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:40.342735 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.339708 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt527\" (UniqueName: \"kubernetes.io/projected/13d58406-ef62-4c3e-a222-91f79c761619-kube-api-access-jt527\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:40.342735 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.339730 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d58406-ef62-4c3e-a222-91f79c761619-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:40.348482 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.348451 4431 generic.go:356] "Generic (PLEG): container finished" podID="13d58406-ef62-4c3e-a222-91f79c761619" containerID="ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf" exitCode=0 Oct 30 07:00:40.348611 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.348497 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" event={"ID":"13d58406-ef62-4c3e-a222-91f79c761619","Type":"ContainerDied","Data":"ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf"} Oct 30 07:00:40.348611 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.348531 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" Oct 30 07:00:40.348722 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.348703 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt" event={"ID":"13d58406-ef62-4c3e-a222-91f79c761619","Type":"ContainerDied","Data":"dc3910cc9c0d00c50bc5dea089a021510110860e26f0c37dc70d02c397c20537"} Oct 30 07:00:40.348779 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.348742 4431 scope.go:122] "RemoveContainer" containerID="ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf" Oct 30 07:00:40.358761 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.358740 4431 scope.go:122] "RemoveContainer" containerID="ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf" Oct 30 07:00:40.359053 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:40.359030 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf\": container with ID starting with ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf not found: ID does not exist" containerID="ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf" Oct 30 07:00:40.359144 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.359058 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf"} err="failed to get container status \"ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf\": rpc error: code = NotFound desc = could not find container \"ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf\": container with ID starting with ae6be901845bffff2f2cada16ee6e4231897853a9501ab44419144b16658f7bf not found: ID does not exist" Oct 30 07:00:40.401285 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.401223 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt"] Oct 30 07:00:40.404755 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:40.404728 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aaf467-69d568c457-kkvbt"] Oct 30 07:00:41.337089 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:41.337054 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d58406-ef62-4c3e-a222-91f79c761619" path="/var/lib/kubelet/pods/13d58406-ef62-4c3e-a222-91f79c761619/volumes" Oct 30 07:00:44.078941 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.078876 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 07:00:44.176038 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.176003 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb21f6c-d934-451c-ae6e-e878af83cc49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb21f6c-d934-451c-ae6e-e878af83cc49-plugins-config-volume\") pod \"0cb21f6c-d934-451c-ae6e-e878af83cc49\" (UID: \"0cb21f6c-d934-451c-ae6e-e878af83cc49\") " Oct 30 07:00:44.176229 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.176142 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb21f6c-d934-451c-ae6e-e878af83cc49-kube-api-access-6z267\" (UniqueName: \"kubernetes.io/projected/0cb21f6c-d934-451c-ae6e-e878af83cc49-kube-api-access-6z267\") pod \"0cb21f6c-d934-451c-ae6e-e878af83cc49\" (UID: \"0cb21f6c-d934-451c-ae6e-e878af83cc49\") " Oct 30 07:00:44.176484 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.176453 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb21f6c-d934-451c-ae6e-e878af83cc49-plugins-config-volume" pod "0cb21f6c-d934-451c-ae6e-e878af83cc49" (UID: "0cb21f6c-d934-451c-ae6e-e878af83cc49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:44.176601 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.176583 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb21f6c-d934-451c-ae6e-e878af83cc49-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:44.178224 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.178192 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb21f6c-d934-451c-ae6e-e878af83cc49-kube-api-access-6z267" pod "0cb21f6c-d934-451c-ae6e-e878af83cc49" (UID: "0cb21f6c-d934-451c-ae6e-e878af83cc49"). InnerVolumeSpecName "kube-api-access-6z267". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:44.277342 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.277292 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z267\" (UniqueName: \"kubernetes.io/projected/0cb21f6c-d934-451c-ae6e-e878af83cc49-kube-api-access-6z267\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:44.369943 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.369861 4431 generic.go:356] "Generic (PLEG): container finished" podID="0cb21f6c-d934-451c-ae6e-e878af83cc49" containerID="1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9" exitCode=0 Oct 30 07:00:44.369943 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.369905 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" Oct 30 07:00:44.370124 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.369951 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" event={"ID":"0cb21f6c-d934-451c-ae6e-e878af83cc49","Type":"ContainerDied","Data":"1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9"} Oct 30 07:00:44.370124 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.370004 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9" event={"ID":"0cb21f6c-d934-451c-ae6e-e878af83cc49","Type":"ContainerDied","Data":"c6ce3942aac27494d2427c39b53b3c52ac02fa76561586b9804d0fe212c2b53f"} Oct 30 07:00:44.370124 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.370033 4431 scope.go:122] "RemoveContainer" containerID="1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9" Oct 30 07:00:44.380678 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.380660 4431 scope.go:122] "RemoveContainer" containerID="1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9" Oct 30 07:00:44.380967 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:44.380949 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9\": container with ID starting with 1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9 not found: ID does not exist" containerID="1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9" Oct 30 07:00:44.381016 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.380977 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9"} err="failed to get container status \"1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9\": rpc error: code = NotFound desc = could not find container \"1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9\": container with ID starting with 1b39d5bba61db30319e7cf8564df4ae08a414a28e90a7dcff9f449dea424d4b9 not found: ID does not exist" Oct 30 07:00:44.419433 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.419411 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9"] Oct 30 07:00:44.428428 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:44.428401 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aaf467-544d8fdc84-jjcv9"] Oct 30 07:00:45.336267 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:45.336230 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb21f6c-d934-451c-ae6e-e878af83cc49" path="/var/lib/kubelet/pods/0cb21f6c-d934-451c-ae6e-e878af83cc49/volumes" Oct 30 07:00:46.600138 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:46.600090 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" podUID="3c5462b6-b727-4176-af24-18f12c02d3c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:46.709770 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:46.709737 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr"] Oct 30 07:00:46.711530 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:46.711499 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" containerID="cri-o://b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406" gracePeriod=30 Oct 30 07:00:46.711636 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:46.709696 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" podUID="2b161474-abc3-429d-9e13-a0fb49f95717" containerName="main" containerID="cri-o://29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522" gracePeriod=45 Oct 30 07:00:46.727088 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:46.727052 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" podUID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" containerName="main" containerID="cri-o://ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5" gracePeriod=45 Oct 30 07:00:46.977802 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:46.977778 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 07:00:47.003726 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.003700 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b161474-abc3-429d-9e13-a0fb49f95717-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b161474-abc3-429d-9e13-a0fb49f95717-tls-certs\") pod \"2b161474-abc3-429d-9e13-a0fb49f95717\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " Oct 30 07:00:47.003929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.003906 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-tmp-dir\") pod \"2b161474-abc3-429d-9e13-a0fb49f95717\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " Oct 30 07:00:47.004014 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.003972 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-home\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-home\") pod \"2b161474-abc3-429d-9e13-a0fb49f95717\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " Oct 30 07:00:47.004863 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.004045 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-model-cache\") pod \"2b161474-abc3-429d-9e13-a0fb49f95717\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " Oct 30 07:00:47.004863 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.004089 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b161474-abc3-429d-9e13-a0fb49f95717-kube-api-access-nrct5\" (UniqueName: \"kubernetes.io/projected/2b161474-abc3-429d-9e13-a0fb49f95717-kube-api-access-nrct5\") pod \"2b161474-abc3-429d-9e13-a0fb49f95717\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " Oct 30 07:00:47.004863 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.004124 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-dshm\") pod \"2b161474-abc3-429d-9e13-a0fb49f95717\" (UID: \"2b161474-abc3-429d-9e13-a0fb49f95717\") " Oct 30 07:00:47.004863 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.004255 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 07:00:47.004863 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.004257 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-home" pod "2b161474-abc3-429d-9e13-a0fb49f95717" (UID: "2b161474-abc3-429d-9e13-a0fb49f95717"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.004863 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.004503 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-model-cache" pod "2b161474-abc3-429d-9e13-a0fb49f95717" (UID: "2b161474-abc3-429d-9e13-a0fb49f95717"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.004863 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.004540 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.004863 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.004781 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-tmp-dir" pod "2b161474-abc3-429d-9e13-a0fb49f95717" (UID: "2b161474-abc3-429d-9e13-a0fb49f95717"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.006055 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.006025 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b161474-abc3-429d-9e13-a0fb49f95717-tls-certs" pod "2b161474-abc3-429d-9e13-a0fb49f95717" (UID: "2b161474-abc3-429d-9e13-a0fb49f95717"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:47.006721 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.006687 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-dshm" pod "2b161474-abc3-429d-9e13-a0fb49f95717" (UID: "2b161474-abc3-429d-9e13-a0fb49f95717"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.006811 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.006774 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b161474-abc3-429d-9e13-a0fb49f95717-kube-api-access-nrct5" pod "2b161474-abc3-429d-9e13-a0fb49f95717" (UID: "2b161474-abc3-429d-9e13-a0fb49f95717"). InnerVolumeSpecName "kube-api-access-nrct5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.105731 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tls-certs\") pod \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.105810 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-model-cache\") pod \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.105865 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-home\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-home\") pod \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.105954 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tmp-dir\") pod \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.105991 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-dshm\") pod \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.106030 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713a2758-f5d8-4a1d-a9b3-3331530ead0a-kube-api-access-q9jv2\" (UniqueName: \"kubernetes.io/projected/713a2758-f5d8-4a1d-a9b3-3331530ead0a-kube-api-access-q9jv2\") pod \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\" (UID: \"713a2758-f5d8-4a1d-a9b3-3331530ead0a\") " Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.106406 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.106430 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrct5\" (UniqueName: \"kubernetes.io/projected/2b161474-abc3-429d-9e13-a0fb49f95717-kube-api-access-nrct5\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.106443 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.106458 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b161474-abc3-429d-9e13-a0fb49f95717-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.106929 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.106472 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b161474-abc3-429d-9e13-a0fb49f95717-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.107746 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.107078 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-home" pod "713a2758-f5d8-4a1d-a9b3-3331530ead0a" (UID: "713a2758-f5d8-4a1d-a9b3-3331530ead0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.111356 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.108731 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tmp-dir" pod "713a2758-f5d8-4a1d-a9b3-3331530ead0a" (UID: "713a2758-f5d8-4a1d-a9b3-3331530ead0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.111356 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.111108 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-dshm" pod "713a2758-f5d8-4a1d-a9b3-3331530ead0a" (UID: "713a2758-f5d8-4a1d-a9b3-3331530ead0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.113591 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.113560 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-model-cache" pod "713a2758-f5d8-4a1d-a9b3-3331530ead0a" (UID: "713a2758-f5d8-4a1d-a9b3-3331530ead0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:47.113707 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.113662 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713a2758-f5d8-4a1d-a9b3-3331530ead0a-kube-api-access-q9jv2" pod "713a2758-f5d8-4a1d-a9b3-3331530ead0a" (UID: "713a2758-f5d8-4a1d-a9b3-3331530ead0a"). InnerVolumeSpecName "kube-api-access-q9jv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:47.115451 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.115422 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tls-certs" pod "713a2758-f5d8-4a1d-a9b3-3331530ead0a" (UID: "713a2758-f5d8-4a1d-a9b3-3331530ead0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:47.206961 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.206927 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tls-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.206961 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.206957 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-model-cache\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.206961 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.206966 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-home\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.207197 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.206974 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-tmp-dir\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.207197 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.206982 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/713a2758-f5d8-4a1d-a9b3-3331530ead0a-dshm\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.207197 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.206990 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9jv2\" (UniqueName: \"kubernetes.io/projected/713a2758-f5d8-4a1d-a9b3-3331530ead0a-kube-api-access-q9jv2\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.389376 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.389082 4431 generic.go:356] "Generic (PLEG): container finished" podID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" containerID="ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5" exitCode=0 Oct 30 07:00:47.389376 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.389127 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" event={"ID":"713a2758-f5d8-4a1d-a9b3-3331530ead0a","Type":"ContainerDied","Data":"ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5"} Oct 30 07:00:47.389376 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.389167 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" Oct 30 07:00:47.389376 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.389186 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524" event={"ID":"713a2758-f5d8-4a1d-a9b3-3331530ead0a","Type":"ContainerDied","Data":"39455ff6986350aee8105ccf65ed593f767564a91bbe8dffa2bf35d5b5da204a"} Oct 30 07:00:47.389376 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.389212 4431 scope.go:122] "RemoveContainer" containerID="ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5" Oct 30 07:00:47.392237 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.392212 4431 generic.go:356] "Generic (PLEG): container finished" podID="3c5462b6-b727-4176-af24-18f12c02d3c2" containerID="3d731db4a89fa9b9efe99eb7f730a10faec5d68ad63f810a680c255e0038f97d" exitCode=0 Oct 30 07:00:47.392421 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.392275 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" event={"ID":"3c5462b6-b727-4176-af24-18f12c02d3c2","Type":"ContainerDied","Data":"3d731db4a89fa9b9efe99eb7f730a10faec5d68ad63f810a680c255e0038f97d"} Oct 30 07:00:47.393924 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.393893 4431 generic.go:356] "Generic (PLEG): container finished" podID="2b161474-abc3-429d-9e13-a0fb49f95717" containerID="29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522" exitCode=0 Oct 30 07:00:47.394034 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.393991 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" event={"ID":"2b161474-abc3-429d-9e13-a0fb49f95717","Type":"ContainerDied","Data":"29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522"} Oct 30 07:00:47.394034 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.394018 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" event={"ID":"2b161474-abc3-429d-9e13-a0fb49f95717","Type":"ContainerDied","Data":"345285ef61b22a981ba708c8965f21cd5f5fdcfe02ec297ec8ebdb3aa618aa9a"} Oct 30 07:00:47.394162 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.394082 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f" Oct 30 07:00:47.400806 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.400786 4431 scope.go:122] "RemoveContainer" containerID="ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5" Oct 30 07:00:47.401613 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:47.401589 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5\": container with ID starting with ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5 not found: ID does not exist" containerID="ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5" Oct 30 07:00:47.401728 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.401624 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5"} err="failed to get container status \"ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5\": rpc error: code = NotFound desc = could not find container \"ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5\": container with ID starting with ec83183de30705f980f8cb8c339be4257415166e450e62dd77fec088ee8699d5 not found: ID does not exist" Oct 30 07:00:47.401728 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.401649 4431 scope.go:122] "RemoveContainer" containerID="29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522" Oct 30 07:00:47.410248 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.410225 4431 scope.go:122] "RemoveContainer" containerID="29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522" Oct 30 07:00:47.410546 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:47.410526 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522\": container with ID starting with 29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522 not found: ID does not exist" containerID="29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522" Oct 30 07:00:47.410618 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.410556 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522"} err="failed to get container status \"29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522\": rpc error: code = NotFound desc = could not find container \"29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522\": container with ID starting with 29cf6a27e22090c794dbdfb7096e0a90ec2a145515ffb564d9bc8e23dcd8a522 not found: ID does not exist" Oct 30 07:00:47.440840 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.440811 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f"] Oct 30 07:00:47.452631 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.452607 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/distinct-w1-8d44dd-kserve-975465b7b-2425f"] Oct 30 07:00:47.460316 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.460251 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 07:00:47.481582 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.481560 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524"] Oct 30 07:00:47.483475 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.483455 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8d44dd/premium-w1-8d44dd-kserve-5ff97bbffc-6x524"] Oct 30 07:00:47.509977 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.509957 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5462b6-b727-4176-af24-18f12c02d3c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5462b6-b727-4176-af24-18f12c02d3c2-plugins-config-volume\") pod \"3c5462b6-b727-4176-af24-18f12c02d3c2\" (UID: \"3c5462b6-b727-4176-af24-18f12c02d3c2\") " Oct 30 07:00:47.510077 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.510011 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5462b6-b727-4176-af24-18f12c02d3c2-kube-api-access-cm44c\" (UniqueName: \"kubernetes.io/projected/3c5462b6-b727-4176-af24-18f12c02d3c2-kube-api-access-cm44c\") pod \"3c5462b6-b727-4176-af24-18f12c02d3c2\" (UID: \"3c5462b6-b727-4176-af24-18f12c02d3c2\") " Oct 30 07:00:47.510314 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.510280 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5462b6-b727-4176-af24-18f12c02d3c2-plugins-config-volume" pod "3c5462b6-b727-4176-af24-18f12c02d3c2" (UID: "3c5462b6-b727-4176-af24-18f12c02d3c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:47.511996 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.511975 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5462b6-b727-4176-af24-18f12c02d3c2-kube-api-access-cm44c" pod "3c5462b6-b727-4176-af24-18f12c02d3c2" (UID: "3c5462b6-b727-4176-af24-18f12c02d3c2"). InnerVolumeSpecName "kube-api-access-cm44c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:47.611206 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.611173 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm44c\" (UniqueName: \"kubernetes.io/projected/3c5462b6-b727-4176-af24-18f12c02d3c2-kube-api-access-cm44c\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:47.611206 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:47.611201 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5462b6-b727-4176-af24-18f12c02d3c2-plugins-config-volume\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:48.401435 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:48.401407 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" Oct 30 07:00:48.401435 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:48.401424 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt" event={"ID":"3c5462b6-b727-4176-af24-18f12c02d3c2","Type":"ContainerDied","Data":"a27a9b9b485e94b64fef07c3a737712e36de584e032b2afd7e90724183c93603"} Oct 30 07:00:48.401694 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:48.401477 4431 scope.go:122] "RemoveContainer" containerID="3d731db4a89fa9b9efe99eb7f730a10faec5d68ad63f810a680c255e0038f97d" Oct 30 07:00:48.451710 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:48.451684 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt"] Oct 30 07:00:48.464686 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:48.464661 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8d6349-6766f4df5b-kw5gt"] Oct 30 07:00:49.337746 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:49.337710 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b161474-abc3-429d-9e13-a0fb49f95717" path="/var/lib/kubelet/pods/2b161474-abc3-429d-9e13-a0fb49f95717/volumes" Oct 30 07:00:49.338262 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:49.338241 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5462b6-b727-4176-af24-18f12c02d3c2" path="/var/lib/kubelet/pods/3c5462b6-b727-4176-af24-18f12c02d3c2/volumes" Oct 30 07:00:49.338809 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:49.338790 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" path="/var/lib/kubelet/pods/713a2758-f5d8-4a1d-a9b3-3331530ead0a/volumes" Oct 30 07:00:51.965085 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:51.965060 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 07:00:52.052698 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.052630 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-envoy\") pod \"50c35553-4ed8-4baf-8106-e21aeb16eae4\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " Oct 30 07:00:52.052865 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.052717 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-certs\") pod \"50c35553-4ed8-4baf-8106-e21aeb16eae4\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " Oct 30 07:00:52.052865 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.052742 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-kube-api-access-k5g9c\") pod \"50c35553-4ed8-4baf-8106-e21aeb16eae4\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " Oct 30 07:00:52.052865 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.052762 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-token\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-token\") pod \"50c35553-4ed8-4baf-8106-e21aeb16eae4\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " Oct 30 07:00:52.052865 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.052807 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c35553-4ed8-4baf-8106-e21aeb16eae4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c35553-4ed8-4baf-8106-e21aeb16eae4-istiod-ca-cert\") pod \"50c35553-4ed8-4baf-8106-e21aeb16eae4\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " Oct 30 07:00:52.052865 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.052835 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-podinfo\") pod \"50c35553-4ed8-4baf-8106-e21aeb16eae4\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " Oct 30 07:00:52.052865 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.052862 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-credential-socket\") pod \"50c35553-4ed8-4baf-8106-e21aeb16eae4\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " Oct 30 07:00:52.053200 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.052891 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-data\") pod \"50c35553-4ed8-4baf-8106-e21aeb16eae4\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " Oct 30 07:00:52.053200 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.052927 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-socket\") pod \"50c35553-4ed8-4baf-8106-e21aeb16eae4\" (UID: \"50c35553-4ed8-4baf-8106-e21aeb16eae4\") " Oct 30 07:00:52.053200 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.053018 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-certs" pod "50c35553-4ed8-4baf-8106-e21aeb16eae4" (UID: "50c35553-4ed8-4baf-8106-e21aeb16eae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:52.053399 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.053257 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.053399 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.053251 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-credential-socket" pod "50c35553-4ed8-4baf-8106-e21aeb16eae4" (UID: "50c35553-4ed8-4baf-8106-e21aeb16eae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:52.053542 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.053511 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-data" pod "50c35553-4ed8-4baf-8106-e21aeb16eae4" (UID: "50c35553-4ed8-4baf-8106-e21aeb16eae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:52.053542 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.053527 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-socket" pod "50c35553-4ed8-4baf-8106-e21aeb16eae4" (UID: "50c35553-4ed8-4baf-8106-e21aeb16eae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:52.053662 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.053520 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c35553-4ed8-4baf-8106-e21aeb16eae4-istiod-ca-cert" pod "50c35553-4ed8-4baf-8106-e21aeb16eae4" (UID: "50c35553-4ed8-4baf-8106-e21aeb16eae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:52.054934 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.054906 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-envoy" pod "50c35553-4ed8-4baf-8106-e21aeb16eae4" (UID: "50c35553-4ed8-4baf-8106-e21aeb16eae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:52.055070 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.055047 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-podinfo" pod "50c35553-4ed8-4baf-8106-e21aeb16eae4" (UID: "50c35553-4ed8-4baf-8106-e21aeb16eae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:00:52.055153 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.055140 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-kube-api-access-k5g9c" pod "50c35553-4ed8-4baf-8106-e21aeb16eae4" (UID: "50c35553-4ed8-4baf-8106-e21aeb16eae4"). InnerVolumeSpecName "kube-api-access-k5g9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:52.055263 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.055246 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-token" pod "50c35553-4ed8-4baf-8106-e21aeb16eae4" (UID: "50c35553-4ed8-4baf-8106-e21aeb16eae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:52.154194 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.154151 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.154194 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.154190 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.154194 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.154200 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-kube-api-access-k5g9c\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.154419 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.154209 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.154419 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.154219 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c35553-4ed8-4baf-8106-e21aeb16eae4-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.154419 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.154226 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.154419 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.154234 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.154419 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.154242 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c35553-4ed8-4baf-8106-e21aeb16eae4-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.294838 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.294798 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp"] Oct 30 07:00:52.295081 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.295055 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" podUID="54384483-f540-422a-bb19-617300c9b0a8" containerName="maas-api" containerID="cri-o://522c2500daef39a312dd4122a56a3158806a7e43783c45a6800322963ac97ad7" gracePeriod=30 Oct 30 07:00:52.424805 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.424776 4431 generic.go:356] "Generic (PLEG): container finished" podID="54384483-f540-422a-bb19-617300c9b0a8" containerID="522c2500daef39a312dd4122a56a3158806a7e43783c45a6800322963ac97ad7" exitCode=0 Oct 30 07:00:52.424966 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.424848 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" event={"ID":"54384483-f540-422a-bb19-617300c9b0a8","Type":"ContainerDied","Data":"522c2500daef39a312dd4122a56a3158806a7e43783c45a6800322963ac97ad7"} Oct 30 07:00:52.426420 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.426396 4431 generic.go:356] "Generic (PLEG): container finished" podID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerID="b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406" exitCode=0 Oct 30 07:00:52.426557 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.426444 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" event={"ID":"50c35553-4ed8-4baf-8106-e21aeb16eae4","Type":"ContainerDied","Data":"b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406"} Oct 30 07:00:52.426557 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.426458 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" Oct 30 07:00:52.426557 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.426495 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr" event={"ID":"50c35553-4ed8-4baf-8106-e21aeb16eae4","Type":"ContainerDied","Data":"45c6260cda03f17a0bad1ec9345881836ee0bba4e759fd94bedb64368de84192"} Oct 30 07:00:52.426557 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.426526 4431 scope.go:122] "RemoveContainer" containerID="b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406" Oct 30 07:00:52.436807 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.436791 4431 scope.go:122] "RemoveContainer" containerID="b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406" Oct 30 07:00:52.437090 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:00:52.437070 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406\": container with ID starting with b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406 not found: ID does not exist" containerID="b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406" Oct 30 07:00:52.437157 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.437102 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406"} err="failed to get container status \"b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406\": rpc error: code = NotFound desc = could not find container \"b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406\": container with ID starting with b4f5a6aad12956b2c7b15b7d3624f9455852e016dcac205322dd1e82a2ea2406 not found: ID does not exist" Oct 30 07:00:52.477110 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.477084 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr"] Oct 30 07:00:52.483547 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.483522 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-w7klr"] Oct 30 07:00:52.530572 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.530539 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 07:00:52.657481 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.657401 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54384483-f540-422a-bb19-617300c9b0a8-kube-api-access-w9m8t\" (UniqueName: \"kubernetes.io/projected/54384483-f540-422a-bb19-617300c9b0a8-kube-api-access-w9m8t\") pod \"54384483-f540-422a-bb19-617300c9b0a8\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " Oct 30 07:00:52.657633 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.657507 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54384483-f540-422a-bb19-617300c9b0a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54384483-f540-422a-bb19-617300c9b0a8-openshift-service-ca\") pod \"54384483-f540-422a-bb19-617300c9b0a8\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " Oct 30 07:00:52.657633 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.657543 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls\") pod \"54384483-f540-422a-bb19-617300c9b0a8\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " Oct 30 07:00:52.657633 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.657567 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-metrics-certs\") pod \"54384483-f540-422a-bb19-617300c9b0a8\" (UID: \"54384483-f540-422a-bb19-617300c9b0a8\") " Oct 30 07:00:52.657960 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.657922 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54384483-f540-422a-bb19-617300c9b0a8-openshift-service-ca" pod "54384483-f540-422a-bb19-617300c9b0a8" (UID: "54384483-f540-422a-bb19-617300c9b0a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:52.659571 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.659537 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54384483-f540-422a-bb19-617300c9b0a8-kube-api-access-w9m8t" pod "54384483-f540-422a-bb19-617300c9b0a8" (UID: "54384483-f540-422a-bb19-617300c9b0a8"). InnerVolumeSpecName "kube-api-access-w9m8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:52.659696 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.659678 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-metrics-certs" pod "54384483-f540-422a-bb19-617300c9b0a8" (UID: "54384483-f540-422a-bb19-617300c9b0a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:52.659751 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.659697 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls" pod "54384483-f540-422a-bb19-617300c9b0a8" (UID: "54384483-f540-422a-bb19-617300c9b0a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:52.758616 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.758590 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9m8t\" (UniqueName: \"kubernetes.io/projected/54384483-f540-422a-bb19-617300c9b0a8-kube-api-access-w9m8t\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.758616 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.758614 4431 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54384483-f540-422a-bb19-617300c9b0a8-openshift-service-ca\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.758787 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.758625 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-maas-api-tls\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:52.758787 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:52.758634 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54384483-f540-422a-bb19-617300c9b0a8-metrics-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:00:53.336113 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:53.336081 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" path="/var/lib/kubelet/pods/50c35553-4ed8-4baf-8106-e21aeb16eae4/volumes" Oct 30 07:00:53.437391 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:53.437358 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" event={"ID":"54384483-f540-422a-bb19-617300c9b0a8","Type":"ContainerDied","Data":"2f16ad1fe4cfdca7a377b3ea1adc01c57637e1789a43afc9b48461c44d60e40e"} Oct 30 07:00:53.437566 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:53.437407 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp" Oct 30 07:00:53.437566 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:53.437422 4431 scope.go:122] "RemoveContainer" containerID="522c2500daef39a312dd4122a56a3158806a7e43783c45a6800322963ac97ad7" Oct 30 07:00:53.490867 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:53.490838 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp"] Oct 30 07:00:53.503023 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:53.502997 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8d44dd-6cccfc67fc-cmshp"] Oct 30 07:00:55.336697 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:00:55.336668 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54384483-f540-422a-bb19-617300c9b0a8" path="/var/lib/kubelet/pods/54384483-f540-422a-bb19-617300c9b0a8/volumes" Oct 30 07:01:00.643166 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:00.643129 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw"] Oct 30 07:01:00.643664 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:00.643426 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" containerID="cri-o://86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2" gracePeriod=30 Oct 30 07:01:01.110711 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:01.110673 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:01:01.110886 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:01.110740 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:01:03.699720 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:01:03.699673 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 07:01:04.785792 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:04.785761 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6577b95894-swx2g_7bfe7f12-7c1e-4e56-8f6d-b94527d0428e/manager/0.log" Oct 30 07:01:05.219010 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.218984 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwq2b_a879b8c8-3bc4-4f44-9691-e924c6b703a8/manager/0.log" Oct 30 07:01:05.885882 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.885858 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 07:01:05.976847 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.976810 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-credential-socket\") pod \"fc39553a-daaf-482c-8be9-476826b80ffe\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " Oct 30 07:01:05.976985 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.976861 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-envoy\") pod \"fc39553a-daaf-482c-8be9-476826b80ffe\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " Oct 30 07:01:05.976985 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.976913 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc39553a-daaf-482c-8be9-476826b80ffe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39553a-daaf-482c-8be9-476826b80ffe-istiod-ca-cert\") pod \"fc39553a-daaf-482c-8be9-476826b80ffe\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " Oct 30 07:01:05.976985 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.976955 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-kube-api-access-pwzs5\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-kube-api-access-pwzs5\") pod \"fc39553a-daaf-482c-8be9-476826b80ffe\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " Oct 30 07:01:05.977173 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.976992 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc39553a-daaf-482c-8be9-476826b80ffe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39553a-daaf-482c-8be9-476826b80ffe-istio-podinfo\") pod \"fc39553a-daaf-482c-8be9-476826b80ffe\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " Oct 30 07:01:05.977173 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977051 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-certs\") pod \"fc39553a-daaf-482c-8be9-476826b80ffe\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " Oct 30 07:01:05.977173 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977064 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-credential-socket" pod "fc39553a-daaf-482c-8be9-476826b80ffe" (UID: "fc39553a-daaf-482c-8be9-476826b80ffe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:01:05.977173 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977129 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-socket\") pod \"fc39553a-daaf-482c-8be9-476826b80ffe\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " Oct 30 07:01:05.977426 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977186 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-data\") pod \"fc39553a-daaf-482c-8be9-476826b80ffe\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " Oct 30 07:01:05.977426 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977231 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-istio-token\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-istio-token\") pod \"fc39553a-daaf-482c-8be9-476826b80ffe\" (UID: \"fc39553a-daaf-482c-8be9-476826b80ffe\") " Oct 30 07:01:05.977541 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977442 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-socket" pod "fc39553a-daaf-482c-8be9-476826b80ffe" (UID: "fc39553a-daaf-482c-8be9-476826b80ffe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:01:05.977595 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977490 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc39553a-daaf-482c-8be9-476826b80ffe-istiod-ca-cert" pod "fc39553a-daaf-482c-8be9-476826b80ffe" (UID: "fc39553a-daaf-482c-8be9-476826b80ffe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:01:05.977651 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977562 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-data" pod "fc39553a-daaf-482c-8be9-476826b80ffe" (UID: "fc39553a-daaf-482c-8be9-476826b80ffe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:01:05.977651 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977612 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-certs" pod "fc39553a-daaf-482c-8be9-476826b80ffe" (UID: "fc39553a-daaf-482c-8be9-476826b80ffe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:01:05.977752 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977669 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:01:05.977752 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977693 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-credential-socket\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:01:05.977752 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.977709 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39553a-daaf-482c-8be9-476826b80ffe-istiod-ca-cert\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:01:05.979127 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.979103 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-envoy" pod "fc39553a-daaf-482c-8be9-476826b80ffe" (UID: "fc39553a-daaf-482c-8be9-476826b80ffe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:01:05.979373 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.979354 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-istio-token" pod "fc39553a-daaf-482c-8be9-476826b80ffe" (UID: "fc39553a-daaf-482c-8be9-476826b80ffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:01:05.979550 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.979533 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-kube-api-access-pwzs5" pod "fc39553a-daaf-482c-8be9-476826b80ffe" (UID: "fc39553a-daaf-482c-8be9-476826b80ffe"). InnerVolumeSpecName "kube-api-access-pwzs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:01:05.979603 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:05.979536 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc39553a-daaf-482c-8be9-476826b80ffe-istio-podinfo" pod "fc39553a-daaf-482c-8be9-476826b80ffe" (UID: "fc39553a-daaf-482c-8be9-476826b80ffe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:01:06.078685 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.078660 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-data\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:01:06.078685 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.078684 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-istio-token\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:01:06.078858 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.078697 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-istio-envoy\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:01:06.078858 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.078709 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwzs5\" (UniqueName: \"kubernetes.io/projected/fc39553a-daaf-482c-8be9-476826b80ffe-kube-api-access-pwzs5\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:01:06.078858 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.078723 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39553a-daaf-482c-8be9-476826b80ffe-istio-podinfo\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:01:06.078858 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.078735 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39553a-daaf-482c-8be9-476826b80ffe-workload-certs\") on node \"ip-10-0-136-123.ec2.internal\" DevicePath \"\"" Oct 30 07:01:06.500211 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.500177 4431 generic.go:356] "Generic (PLEG): container finished" podID="fc39553a-daaf-482c-8be9-476826b80ffe" containerID="86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2" exitCode=0 Oct 30 07:01:06.500424 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.500223 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" Oct 30 07:01:06.500424 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.500245 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" event={"ID":"fc39553a-daaf-482c-8be9-476826b80ffe","Type":"ContainerDied","Data":"86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2"} Oct 30 07:01:06.500424 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.500296 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw" event={"ID":"fc39553a-daaf-482c-8be9-476826b80ffe","Type":"ContainerDied","Data":"addda89392a9f3b777500c9fe141fcec37e0d6aca7e314ddbb1bc4fa09feba8c"} Oct 30 07:01:06.500424 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.500347 4431 scope.go:122] "RemoveContainer" containerID="86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2" Oct 30 07:01:06.509734 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.509713 4431 scope.go:122] "RemoveContainer" containerID="86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2" Oct 30 07:01:06.509994 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:01:06.509976 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2\": container with ID starting with 86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2 not found: ID does not exist" containerID="86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2" Oct 30 07:01:06.510105 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.510002 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2"} err="failed to get container status \"86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2\": rpc error: code = NotFound desc = could not find container \"86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2\": container with ID starting with 86f78affe82ee907ae1ced45fe60570dba7e321daf3dd467b7adcf5357cbacd2 not found: ID does not exist" Oct 30 07:01:06.611948 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.611922 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw"] Oct 30 07:01:06.645810 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:06.645790 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wqpnw"] Oct 30 07:01:07.336937 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:07.336902 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" path="/var/lib/kubelet/pods/fc39553a-daaf-482c-8be9-476826b80ffe/volumes" Oct 30 07:01:18.790084 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:18.790051 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6577b95894-swx2g_7bfe7f12-7c1e-4e56-8f6d-b94527d0428e/manager/0.log" Oct 30 07:01:19.229297 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:19.229267 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwq2b_a879b8c8-3bc4-4f44-9691-e924c6b703a8/manager/0.log" Oct 30 07:01:21.560287 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:21.560255 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-897bbbb96-r6dwj_1aa01b8a-fe13-40b6-8dfc-3d9ba9cf4744/maas-api/0.log" Oct 30 07:01:29.488730 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:29.488695 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c4bb65c68-jwxlp_d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a/kube-auth-proxy/0.log" Oct 30 07:01:30.718633 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:30.718569 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-f26kj_86cdedee-43d9-421a-9ffd-cfc6775eed66/payload-pre-processing/0.log" Oct 30 07:01:30.926898 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:30.926834 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hpphs_9b2cce36-89ca-4ea5-a488-be5af53dbf14/payload-processing/0.log" Oct 30 07:01:31.959220 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:31.959192 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-f6hfd_322f9a55-29c0-4cfb-8f2f-a736321f7b45/main/0.log" Oct 30 07:01:32.156528 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:32.156502 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-q92kk_642ea77d-6dc0-4e6e-9a79-d5056b3d1589/main/0.log" Oct 30 07:01:32.358383 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:32.358295 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ds65s_fd9f37fd-8bf5-4c25-be74-c36945d782eb/main/0.log" Oct 30 07:01:32.562065 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:32.562034 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-l7s2w_292e4b1e-2d8a-42a0-b26c-b4de0c2e44f8/main/0.log" Oct 30 07:01:32.751041 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:32.751010 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfhgh_bf2fe315-4bfd-436b-849e-ce063c496de2/main/0.log" Oct 30 07:01:33.324145 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:33.324118 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-47skp_4199dace-66ce-4982-9ff6-629345b077ba/main/0.log" Oct 30 07:01:33.699509 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:01:33.699476 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 07:01:43.657868 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.657836 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dc77r/must-gather-dc2bl"] Oct 30 07:01:43.658319 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658235 4431 cpu_manager.go:414] "containerMap: removing container" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" Oct 30 07:01:43.658319 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658254 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" Oct 30 07:01:43.658319 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658313 4431 cpu_manager.go:414] "containerMap: removing container" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" Oct 30 07:01:43.658319 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658320 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658327 4431 cpu_manager.go:414] "containerMap: removing container" podUID="95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" containerName="payload-processing" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658333 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" containerName="payload-processing" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658347 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" containerName="main" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658351 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" containerName="main" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658356 4431 cpu_manager.go:414] "containerMap: removing container" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658361 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658367 4431 cpu_manager.go:414] "containerMap: removing container" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658372 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658377 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2d53b413-657e-4353-b14e-75ccd742e78b" containerName="payload-pre-processing" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658381 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d53b413-657e-4353-b14e-75ccd742e78b" containerName="payload-pre-processing" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658390 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658395 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658401 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658406 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658413 4431 cpu_manager.go:414] "containerMap: removing container" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658417 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658422 4431 cpu_manager.go:414] "containerMap: removing container" podUID="918cfec5-83fd-469d-8a53-63990ff9376a" containerName="payload-processing" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658426 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918cfec5-83fd-469d-8a53-63990ff9376a" containerName="payload-processing" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658439 4431 cpu_manager.go:414] "containerMap: removing container" podUID="df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" containerName="maas-api" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658444 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" containerName="maas-api" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658451 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2b161474-abc3-429d-9e13-a0fb49f95717" containerName="main" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658456 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b161474-abc3-429d-9e13-a0fb49f95717" containerName="main" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658462 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658466 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658478 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658483 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658489 4431 cpu_manager.go:414] "containerMap: removing container" podUID="03aa02fa-ef29-4efb-ae52-5ed210c4a404" containerName="payload-pre-processing" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658493 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aa02fa-ef29-4efb-ae52-5ed210c4a404" containerName="payload-pre-processing" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658500 4431 cpu_manager.go:414] "containerMap: removing container" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" Oct 30 07:01:43.658491 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658505 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658512 4431 cpu_manager.go:414] "containerMap: removing container" podUID="9c4debb0-8735-48b2-a4fd-2bb725083af0" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658516 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4debb0-8735-48b2-a4fd-2bb725083af0" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658524 4431 cpu_manager.go:414] "containerMap: removing container" podUID="96a04f24-d9ec-472f-8ec2-002b5cabf3da" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658528 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a04f24-d9ec-472f-8ec2-002b5cabf3da" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658535 4431 cpu_manager.go:414] "containerMap: removing container" podUID="4b4385bc-6024-4651-8422-f785697b8049" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658539 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4385bc-6024-4651-8422-f785697b8049" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658545 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f5b9d11a-e96d-4733-b920-2ecf04bee128" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658549 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b9d11a-e96d-4733-b920-2ecf04bee128" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658556 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658560 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658566 4431 cpu_manager.go:414] "containerMap: removing container" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658570 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658577 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a2c503db-91d1-47a9-a755-72fa60146690" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658581 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c503db-91d1-47a9-a755-72fa60146690" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658586 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f8efc049-5c75-4627-b33c-b0407d1a2067" containerName="payload-pre-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658590 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8efc049-5c75-4627-b33c-b0407d1a2067" containerName="payload-pre-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658598 4431 cpu_manager.go:414] "containerMap: removing container" podUID="0f0c53ac-dd7d-4ee0-b707-51857ccf2724" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658601 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0c53ac-dd7d-4ee0-b707-51857ccf2724" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658607 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658612 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658617 4431 cpu_manager.go:414] "containerMap: removing container" podUID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658622 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658633 4431 cpu_manager.go:414] "containerMap: removing container" podUID="54384483-f540-422a-bb19-617300c9b0a8" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658637 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54384483-f540-422a-bb19-617300c9b0a8" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658643 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658648 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658655 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658659 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658665 4431 cpu_manager.go:414] "containerMap: removing container" podUID="80db6c09-1940-4274-a21c-57ab7dd6738c" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658669 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80db6c09-1940-4274-a21c-57ab7dd6738c" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658674 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8a59b404-db18-4117-a038-181392be01d6" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658679 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a59b404-db18-4117-a038-181392be01d6" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658687 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1d337a85-2994-4119-8d76-c0bbaca18d5e" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658692 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d337a85-2994-4119-8d76-c0bbaca18d5e" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658698 4431 cpu_manager.go:414] "containerMap: removing container" podUID="0cb21f6c-d934-451c-ae6e-e878af83cc49" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658702 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb21f6c-d934-451c-ae6e-e878af83cc49" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658707 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6829b5b1-5091-4483-be7f-dd73b6a4428e" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658712 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6829b5b1-5091-4483-be7f-dd73b6a4428e" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658719 4431 cpu_manager.go:414] "containerMap: removing container" podUID="71c07881-c3bc-48f0-aff4-a4f20e84f387" containerName="discovery" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658723 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c07881-c3bc-48f0-aff4-a4f20e84f387" containerName="discovery" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658733 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b81c0402-ae16-458d-bdae-659aa77365d9" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658737 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81c0402-ae16-458d-bdae-659aa77365d9" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658743 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3bdd9574-f2ab-449e-916e-74fd2f3cda2d" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658748 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bdd9574-f2ab-449e-916e-74fd2f3cda2d" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658753 4431 cpu_manager.go:414] "containerMap: removing container" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658758 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658766 4431 cpu_manager.go:414] "containerMap: removing container" podUID="9d96bcc0-783d-466d-9f38-820842153c58" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658770 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d96bcc0-783d-466d-9f38-820842153c58" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658776 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658781 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658786 4431 cpu_manager.go:414] "containerMap: removing container" podUID="7376eddd-dd2f-4c7a-af89-7c9fb70abf70" containerName="payload-pre-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658791 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7376eddd-dd2f-4c7a-af89-7c9fb70abf70" containerName="payload-pre-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658798 4431 cpu_manager.go:414] "containerMap: removing container" podUID="13d58406-ef62-4c3e-a222-91f79c761619" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658802 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d58406-ef62-4c3e-a222-91f79c761619" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658809 4431 cpu_manager.go:414] "containerMap: removing container" podUID="62504278-ebeb-48fd-a9a3-6a68dff619e4" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658814 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62504278-ebeb-48fd-a9a3-6a68dff619e4" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658820 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3c5462b6-b727-4176-af24-18f12c02d3c2" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658824 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5462b6-b727-4176-af24-18f12c02d3c2" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658896 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d58406-ef62-4c3e-a222-91f79c761619" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658903 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="62504278-ebeb-48fd-a9a3-6a68dff619e4" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658909 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5462b6-b727-4176-af24-18f12c02d3c2" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658915 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658920 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="253e2111-a9ac-4ddf-a3f8-6e86965c99aa" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658926 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c2c9b0-53dd-4f84-bc20-fc34e7aaf4bb" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658931 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64c38d6-979d-4e74-b2ea-ba1d34ccecaf" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658936 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658941 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c35553-4ed8-4baf-8106-e21aeb16eae4" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658946 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d53b413-657e-4353-b14e-75ccd742e78b" containerName="payload-pre-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658953 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658958 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a20d2777-0ed1-4ead-a144-799c102a6d97" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658964 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="54aa0ffd-3489-4d17-9171-f56a8664045c" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658970 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="918cfec5-83fd-469d-8a53-63990ff9376a" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658976 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b161474-abc3-429d-9e13-a0fb49f95717" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658981 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9fb6fa-fbde-4cc5-b43d-8856815fb3cb" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658986 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc39553a-daaf-482c-8be9-476826b80ffe" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658991 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c977e328-7791-4981-82c6-0e48d9e19b83" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.658997 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="03aa02fa-ef29-4efb-ae52-5ed210c4a404" containerName="payload-pre-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659002 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="eecc4014-0448-4d17-a646-acd40564b5cc" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659006 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4debb0-8735-48b2-a4fd-2bb725083af0" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659012 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a04f24-d9ec-472f-8ec2-002b5cabf3da" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659016 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4385bc-6024-4651-8422-f785697b8049" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659020 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b9d11a-e96d-4733-b920-2ecf04bee128" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659025 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659030 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd098f33-cfd7-4505-8abd-0100edc4feef" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659034 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c503db-91d1-47a9-a755-72fa60146690" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659038 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8efc049-5c75-4627-b33c-b0407d1a2067" containerName="payload-pre-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659043 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0c53ac-dd7d-4ee0-b707-51857ccf2724" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659049 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="713a2758-f5d8-4a1d-a9b3-3331530ead0a" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659054 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="54384483-f540-422a-bb19-617300c9b0a8" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659059 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ae7da1-de6f-4171-abcd-60c8ee6c9bdc" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659063 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8505f718-e49c-4231-91f0-20dd4a1be595" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659068 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="80db6c09-1940-4274-a21c-57ab7dd6738c" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659073 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a59b404-db18-4117-a038-181392be01d6" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659078 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d337a85-2994-4119-8d76-c0bbaca18d5e" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659083 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb21f6c-d934-451c-ae6e-e878af83cc49" containerName="payload-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659088 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c07881-c3bc-48f0-aff4-a4f20e84f387" containerName="discovery" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659092 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6829b5b1-5091-4483-be7f-dd73b6a4428e" containerName="main" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659097 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81c0402-ae16-458d-bdae-659aa77365d9" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659101 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bdd9574-f2ab-449e-916e-74fd2f3cda2d" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659106 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="48caab37-f6bb-49eb-8ac9-1390b217079f" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659111 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d96bcc0-783d-466d-9f38-820842153c58" containerName="maas-api" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659115 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7376eddd-dd2f-4c7a-af89-7c9fb70abf70" containerName="payload-pre-processing" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659212 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d865d5a-0560-4d6e-ae24-6985104463f8" containerName="istio-proxy" Oct 30 07:01:43.659458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.659220 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db731cb-7cbb-4aef-9ef2-820c21f6317d" containerName="istio-proxy" Oct 30 07:01:43.662886 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.662251 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dc77r/must-gather-dc2bl" Oct 30 07:01:43.683839 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.683810 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dc77r/must-gather-dc2bl"] Oct 30 07:01:43.693813 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.693790 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dc77r\"/\"default-dockercfg-xtxkj\"" Oct 30 07:01:43.693913 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.693830 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dc77r\"/\"openshift-service-ca.crt\"" Oct 30 07:01:43.693954 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.693917 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dc77r\"/\"kube-root-ca.crt\"" Oct 30 07:01:43.750835 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.750808 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vn9kw/must-gather-ds9cx"] Oct 30 07:01:43.754469 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.754449 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5nqnt/must-gather-vpp6n"] Oct 30 07:01:43.754591 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.754573 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vn9kw/must-gather-ds9cx" Oct 30 07:01:43.757928 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.757907 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nqnt/must-gather-vpp6n" Oct 30 07:01:43.772365 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.772341 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vn9kw\"/\"kube-root-ca.crt\"" Oct 30 07:01:43.772744 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.772428 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vn9kw\"/\"openshift-service-ca.crt\"" Oct 30 07:01:43.774366 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.774344 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vn9kw/must-gather-ds9cx"] Oct 30 07:01:43.775526 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.775503 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnrj6\" (UniqueName: \"kubernetes.io/projected/92edd4a0-f2d0-4236-9257-d53c661c63ad-kube-api-access-cnrj6\") pod \"must-gather-dc2bl\" (UID: \"92edd4a0-f2d0-4236-9257-d53c661c63ad\") " pod="openshift-must-gather-dc77r/must-gather-dc2bl" Oct 30 07:01:43.775697 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.775582 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92edd4a0-f2d0-4236-9257-d53c661c63ad-must-gather-output\") pod \"must-gather-dc2bl\" (UID: \"92edd4a0-f2d0-4236-9257-d53c661c63ad\") " pod="openshift-must-gather-dc77r/must-gather-dc2bl" Oct 30 07:01:43.777112 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.777093 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nqnt/must-gather-vpp6n"] Oct 30 07:01:43.782396 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.782369 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5nqnt\"/\"kube-root-ca.crt\"" Oct 30 07:01:43.782653 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.782617 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5nqnt\"/\"openshift-service-ca.crt\"" Oct 30 07:01:43.782653 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.782637 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vn9kw\"/\"default-dockercfg-zkkjb\"" Oct 30 07:01:43.782653 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.782639 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5nqnt\"/\"default-dockercfg-w6ccv\"" Oct 30 07:01:43.876319 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.876279 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92edd4a0-f2d0-4236-9257-d53c661c63ad-must-gather-output\") pod \"must-gather-dc2bl\" (UID: \"92edd4a0-f2d0-4236-9257-d53c661c63ad\") " pod="openshift-must-gather-dc77r/must-gather-dc2bl" Oct 30 07:01:43.876524 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.876368 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c30b4622-e383-4002-adcc-2acd4ae44db6-must-gather-output\") pod \"must-gather-vpp6n\" (UID: \"c30b4622-e383-4002-adcc-2acd4ae44db6\") " pod="openshift-must-gather-5nqnt/must-gather-vpp6n" Oct 30 07:01:43.876524 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.876415 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tzjb\" (UniqueName: \"kubernetes.io/projected/c30b4622-e383-4002-adcc-2acd4ae44db6-kube-api-access-6tzjb\") pod \"must-gather-vpp6n\" (UID: \"c30b4622-e383-4002-adcc-2acd4ae44db6\") " pod="openshift-must-gather-5nqnt/must-gather-vpp6n" Oct 30 07:01:43.876524 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.876494 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnrj6\" (UniqueName: \"kubernetes.io/projected/92edd4a0-f2d0-4236-9257-d53c661c63ad-kube-api-access-cnrj6\") pod \"must-gather-dc2bl\" (UID: \"92edd4a0-f2d0-4236-9257-d53c661c63ad\") " pod="openshift-must-gather-dc77r/must-gather-dc2bl" Oct 30 07:01:43.876631 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.876555 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcwmf\" (UniqueName: \"kubernetes.io/projected/06f0df8e-5307-446f-b6ba-8dc1a4853a9a-kube-api-access-tcwmf\") pod \"must-gather-ds9cx\" (UID: \"06f0df8e-5307-446f-b6ba-8dc1a4853a9a\") " pod="openshift-must-gather-vn9kw/must-gather-ds9cx" Oct 30 07:01:43.876631 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.876588 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/06f0df8e-5307-446f-b6ba-8dc1a4853a9a-must-gather-output\") pod \"must-gather-ds9cx\" (UID: \"06f0df8e-5307-446f-b6ba-8dc1a4853a9a\") " pod="openshift-must-gather-vn9kw/must-gather-ds9cx" Oct 30 07:01:43.876721 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.876651 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92edd4a0-f2d0-4236-9257-d53c661c63ad-must-gather-output\") pod \"must-gather-dc2bl\" (UID: \"92edd4a0-f2d0-4236-9257-d53c661c63ad\") " pod="openshift-must-gather-dc77r/must-gather-dc2bl" Oct 30 07:01:43.899683 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.899660 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnrj6\" (UniqueName: \"kubernetes.io/projected/92edd4a0-f2d0-4236-9257-d53c661c63ad-kube-api-access-cnrj6\") pod \"must-gather-dc2bl\" (UID: \"92edd4a0-f2d0-4236-9257-d53c661c63ad\") " pod="openshift-must-gather-dc77r/must-gather-dc2bl" Oct 30 07:01:43.976067 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.976043 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dc77r/must-gather-dc2bl" Oct 30 07:01:43.977371 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.977349 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tzjb\" (UniqueName: \"kubernetes.io/projected/c30b4622-e383-4002-adcc-2acd4ae44db6-kube-api-access-6tzjb\") pod \"must-gather-vpp6n\" (UID: \"c30b4622-e383-4002-adcc-2acd4ae44db6\") " pod="openshift-must-gather-5nqnt/must-gather-vpp6n" Oct 30 07:01:43.977433 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.977397 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcwmf\" (UniqueName: \"kubernetes.io/projected/06f0df8e-5307-446f-b6ba-8dc1a4853a9a-kube-api-access-tcwmf\") pod \"must-gather-ds9cx\" (UID: \"06f0df8e-5307-446f-b6ba-8dc1a4853a9a\") " pod="openshift-must-gather-vn9kw/must-gather-ds9cx" Oct 30 07:01:43.977433 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.977423 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/06f0df8e-5307-446f-b6ba-8dc1a4853a9a-must-gather-output\") pod \"must-gather-ds9cx\" (UID: \"06f0df8e-5307-446f-b6ba-8dc1a4853a9a\") " pod="openshift-must-gather-vn9kw/must-gather-ds9cx" Oct 30 07:01:43.977507 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.977476 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c30b4622-e383-4002-adcc-2acd4ae44db6-must-gather-output\") pod \"must-gather-vpp6n\" (UID: \"c30b4622-e383-4002-adcc-2acd4ae44db6\") " pod="openshift-must-gather-5nqnt/must-gather-vpp6n" Oct 30 07:01:43.977787 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.977768 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/06f0df8e-5307-446f-b6ba-8dc1a4853a9a-must-gather-output\") pod \"must-gather-ds9cx\" (UID: \"06f0df8e-5307-446f-b6ba-8dc1a4853a9a\") " pod="openshift-must-gather-vn9kw/must-gather-ds9cx" Oct 30 07:01:43.977824 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:43.977796 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c30b4622-e383-4002-adcc-2acd4ae44db6-must-gather-output\") pod \"must-gather-vpp6n\" (UID: \"c30b4622-e383-4002-adcc-2acd4ae44db6\") " pod="openshift-must-gather-5nqnt/must-gather-vpp6n" Oct 30 07:01:44.000943 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.000918 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tzjb\" (UniqueName: \"kubernetes.io/projected/c30b4622-e383-4002-adcc-2acd4ae44db6-kube-api-access-6tzjb\") pod \"must-gather-vpp6n\" (UID: \"c30b4622-e383-4002-adcc-2acd4ae44db6\") " pod="openshift-must-gather-5nqnt/must-gather-vpp6n" Oct 30 07:01:44.001690 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.001665 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcwmf\" (UniqueName: \"kubernetes.io/projected/06f0df8e-5307-446f-b6ba-8dc1a4853a9a-kube-api-access-tcwmf\") pod \"must-gather-ds9cx\" (UID: \"06f0df8e-5307-446f-b6ba-8dc1a4853a9a\") " pod="openshift-must-gather-vn9kw/must-gather-ds9cx" Oct 30 07:01:44.069713 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.069680 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vn9kw/must-gather-ds9cx" Oct 30 07:01:44.075846 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.075830 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nqnt/must-gather-vpp6n" Oct 30 07:01:44.280191 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.280171 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dc77r/must-gather-dc2bl"] Oct 30 07:01:44.282249 ip-10-0-136-123 kubenswrapper[4431]: W1030 07:01:44.282220 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92edd4a0_f2d0_4236_9257_d53c661c63ad.slice/crio-9b90ef4d8ee5a810ff7cf95a7a2d734667f3e4ffb24c26cdbfaf033f31e465c3 WatchSource:0}: Error finding container 9b90ef4d8ee5a810ff7cf95a7a2d734667f3e4ffb24c26cdbfaf033f31e465c3: Status 404 returned error can't find the container with id 9b90ef4d8ee5a810ff7cf95a7a2d734667f3e4ffb24c26cdbfaf033f31e465c3 Oct 30 07:01:44.283983 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.283965 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:01:44.383778 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.383755 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vn9kw/must-gather-ds9cx"] Oct 30 07:01:44.385718 ip-10-0-136-123 kubenswrapper[4431]: W1030 07:01:44.385692 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f0df8e_5307_446f_b6ba_8dc1a4853a9a.slice/crio-57452685b62a7ee5c6e8efe22f39d06a45a654db73409fc986aed63a41e4a362 WatchSource:0}: Error finding container 57452685b62a7ee5c6e8efe22f39d06a45a654db73409fc986aed63a41e4a362: Status 404 returned error can't find the container with id 57452685b62a7ee5c6e8efe22f39d06a45a654db73409fc986aed63a41e4a362 Oct 30 07:01:44.393646 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.393606 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nqnt/must-gather-vpp6n"] Oct 30 07:01:44.395909 ip-10-0-136-123 kubenswrapper[4431]: W1030 07:01:44.395887 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc30b4622_e383_4002_adcc_2acd4ae44db6.slice/crio-85eabc5af0da7b71876012af12950f70a2011d0b435f49eb43322a17d9b624f4 WatchSource:0}: Error finding container 85eabc5af0da7b71876012af12950f70a2011d0b435f49eb43322a17d9b624f4: Status 404 returned error can't find the container with id 85eabc5af0da7b71876012af12950f70a2011d0b435f49eb43322a17d9b624f4 Oct 30 07:01:44.660364 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.660276 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vn9kw/must-gather-ds9cx" event={"ID":"06f0df8e-5307-446f-b6ba-8dc1a4853a9a","Type":"ContainerStarted","Data":"57452685b62a7ee5c6e8efe22f39d06a45a654db73409fc986aed63a41e4a362"} Oct 30 07:01:44.661421 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.661391 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nqnt/must-gather-vpp6n" event={"ID":"c30b4622-e383-4002-adcc-2acd4ae44db6","Type":"ContainerStarted","Data":"85eabc5af0da7b71876012af12950f70a2011d0b435f49eb43322a17d9b624f4"} Oct 30 07:01:44.662421 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:44.662401 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dc77r/must-gather-dc2bl" event={"ID":"92edd4a0-f2d0-4236-9257-d53c661c63ad","Type":"ContainerStarted","Data":"9b90ef4d8ee5a810ff7cf95a7a2d734667f3e4ffb24c26cdbfaf033f31e465c3"} Oct 30 07:01:45.675829 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:45.674994 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nqnt/must-gather-vpp6n" event={"ID":"c30b4622-e383-4002-adcc-2acd4ae44db6","Type":"ContainerStarted","Data":"5e1a7860761dd5900946718fd14340a75e5607223161bad9bec7c0b04bd326bc"} Oct 30 07:01:46.685582 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:46.685540 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nqnt/must-gather-vpp6n" event={"ID":"c30b4622-e383-4002-adcc-2acd4ae44db6","Type":"ContainerStarted","Data":"2aaa73d89ba2e9a5090207c8792ed43c41113c80d174cbfeef761442e896a629"} Oct 30 07:01:46.741284 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:46.741227 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5nqnt/must-gather-vpp6n" podStartSLOduration=2.868794614 podStartE2EDuration="3.741212881s" totalImagesPullingTime="872.418267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:46.738572764 +0000 UTC m=+4115.545094514" watchObservedRunningTime="2026-10-30 07:01:46.741212881 +0000 UTC m=+4115.547734629" Oct 30 07:01:49.374159 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:49.374113 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lbqgc_9543ca89-5c01-4c07-9992-534b4955e331/global-pull-secret-syncer/0.log" Oct 30 07:01:50.559322 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:50.559260 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kf896_a975ec4d-2d0d-4c74-ba0b-56a9dee81442/konnectivity-agent/0.log" Oct 30 07:01:51.033683 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:51.033565 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-123.ec2.internal_cc0cc40e9c1a761dcecd1aa2a7591d53/haproxy/0.log" Oct 30 07:01:56.826325 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:56.824769 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dc77r/must-gather-dc2bl" event={"ID":"92edd4a0-f2d0-4236-9257-d53c661c63ad","Type":"ContainerStarted","Data":"02b88eccb46f2b6d87d8b0a14ae3c0ae070fa6166622e0cde57ae3133df94017"} Oct 30 07:01:56.826325 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:56.824827 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dc77r/must-gather-dc2bl" event={"ID":"92edd4a0-f2d0-4236-9257-d53c661c63ad","Type":"ContainerStarted","Data":"82aab431da37eb49b5edc38681c62d1a5f819a0d6dc1cb86a3da86cb4ac4c797"} Oct 30 07:01:56.869726 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:56.869556 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dc77r/must-gather-dc2bl" podStartSLOduration=2.50448999 podStartE2EDuration="13.869535282s" totalImagesPullingTime="11.365045292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:56.868519984 +0000 UTC m=+4125.675041735" watchObservedRunningTime="2026-10-30 07:01:56.869535282 +0000 UTC m=+4125.676057031" Oct 30 07:01:58.852585 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:58.852033 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vn9kw/must-gather-ds9cx" event={"ID":"06f0df8e-5307-446f-b6ba-8dc1a4853a9a","Type":"ContainerStarted","Data":"4f157f27b629f913a8b4b5e0be322ecbfff939a4beb17926fcca46a2521f102e"} Oct 30 07:01:58.852585 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:58.852088 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vn9kw/must-gather-ds9cx" event={"ID":"06f0df8e-5307-446f-b6ba-8dc1a4853a9a","Type":"ContainerStarted","Data":"4b72d0c1269ab2ba22ca63d9a2fa02202e996fea9e9d68e6a392b10fbf517912"} Oct 30 07:01:58.876403 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:58.876368 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6577b95894-swx2g_7bfe7f12-7c1e-4e56-8f6d-b94527d0428e/manager/0.log" Oct 30 07:01:58.906775 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:58.906702 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vn9kw/must-gather-ds9cx" podStartSLOduration=2.114009031 podStartE2EDuration="15.906682862s" totalImagesPullingTime="13.792673831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:58.904613681 +0000 UTC m=+4127.711135430" watchObservedRunningTime="2026-10-30 07:01:58.906682862 +0000 UTC m=+4127.713204612" Oct 30 07:01:59.125056 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:59.124966 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwq2b_a879b8c8-3bc4-4f44-9691-e924c6b703a8/manager/0.log" Oct 30 07:01:59.865537 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:59.865502 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv"] Oct 30 07:01:59.871743 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:59.871714 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:01:59.892802 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:59.892720 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv"] Oct 30 07:01:59.951758 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:59.951718 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-sys\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:01:59.952376 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:59.952344 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-lib-modules\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:01:59.952553 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:59.952438 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-podres\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:01:59.952553 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:59.952543 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-proc\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:01:59.953224 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:01:59.952949 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xbh\" (UniqueName: \"kubernetes.io/projected/f1a027b9-a06b-47fc-9829-1198f2d1c690-kube-api-access-b4xbh\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.056341 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.054787 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-sys\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.056341 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.054837 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-lib-modules\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.056341 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.054866 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-podres\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.056341 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.054889 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-proc\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.056341 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.054933 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xbh\" (UniqueName: \"kubernetes.io/projected/f1a027b9-a06b-47fc-9829-1198f2d1c690-kube-api-access-b4xbh\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.056341 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.055359 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-sys\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.056341 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.055449 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-lib-modules\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.056341 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.055507 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-podres\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.056341 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.055553 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1a027b9-a06b-47fc-9829-1198f2d1c690-proc\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.078575 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.078544 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xbh\" (UniqueName: \"kubernetes.io/projected/f1a027b9-a06b-47fc-9829-1198f2d1c690-kube-api-access-b4xbh\") pod \"perf-node-gather-daemonset-pjrmv\" (UID: \"f1a027b9-a06b-47fc-9829-1198f2d1c690\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.195827 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.195786 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.522044 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.521965 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv"] Oct 30 07:02:00.550077 ip-10-0-136-123 kubenswrapper[4431]: W1030 07:02:00.549820 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1a027b9_a06b_47fc_9829_1198f2d1c690.slice/crio-7657141f0f1dc14b5962eec81b4e37edf7b388320ddc830c4bd688167a10814a WatchSource:0}: Error finding container 7657141f0f1dc14b5962eec81b4e37edf7b388320ddc830c4bd688167a10814a: Status 404 returned error can't find the container with id 7657141f0f1dc14b5962eec81b4e37edf7b388320ddc830c4bd688167a10814a Oct 30 07:02:00.863894 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.863720 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6577b95894-swx2g_7bfe7f12-7c1e-4e56-8f6d-b94527d0428e/manager/0.log" Oct 30 07:02:00.866006 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.865961 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" event={"ID":"f1a027b9-a06b-47fc-9829-1198f2d1c690","Type":"ContainerStarted","Data":"ca1f79ca6d8bab3dde4a8052e606c04517264416616d3f15fce66e4a6ad7aaab"} Oct 30 07:02:00.866470 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.866020 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" event={"ID":"f1a027b9-a06b-47fc-9829-1198f2d1c690","Type":"ContainerStarted","Data":"7657141f0f1dc14b5962eec81b4e37edf7b388320ddc830c4bd688167a10814a"} Oct 30 07:02:00.866470 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.866105 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:00.906533 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:00.906477 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" podStartSLOduration=1.9064626850000002 podStartE2EDuration="1.906462685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 07:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:00.904899799 +0000 UTC m=+4129.711421559" watchObservedRunningTime="2026-10-30 07:02:00.906462685 +0000 UTC m=+4129.712984432" Oct 30 07:02:01.038935 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:01.038898 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwq2b_a879b8c8-3bc4-4f44-9691-e924c6b703a8/manager/0.log" Oct 30 07:02:02.135280 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:02.135236 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6577b95894-swx2g_7bfe7f12-7c1e-4e56-8f6d-b94527d0428e/manager/0.log" Oct 30 07:02:02.289583 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:02.289545 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwq2b_a879b8c8-3bc4-4f44-9691-e924c6b703a8/manager/0.log" Oct 30 07:02:03.427514 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:03.427476 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-92tw7_68e9dfda-59df-4877-976a-3f3ce8019df1/node-exporter/0.log" Oct 30 07:02:03.553284 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:03.553228 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-92tw7_68e9dfda-59df-4877-976a-3f3ce8019df1/kube-rbac-proxy/0.log" Oct 30 07:02:03.673625 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:03.673563 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-92tw7_68e9dfda-59df-4877-976a-3f3ce8019df1/init-textfile/0.log" Oct 30 07:02:03.708347 ip-10-0-136-123 kubenswrapper[4431]: E1030 07:02:03.708270 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rmk2m" containerName="extractor" Oct 30 07:02:03.750109 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:03.750055 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6577b95894-swx2g_7bfe7f12-7c1e-4e56-8f6d-b94527d0428e/manager/0.log" Oct 30 07:02:03.909789 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:03.909761 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwq2b_a879b8c8-3bc4-4f44-9691-e924c6b703a8/manager/0.log" Oct 30 07:02:04.922817 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:04.922788 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6577b95894-swx2g_7bfe7f12-7c1e-4e56-8f6d-b94527d0428e/manager/0.log" Oct 30 07:02:05.076545 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:05.076490 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwq2b_a879b8c8-3bc4-4f44-9691-e924c6b703a8/manager/0.log" Oct 30 07:02:05.863584 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:05.863544 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-zzn7z_808a129f-fd2d-475c-b70e-41bf5695490b/prometheus-operator/0.log" Oct 30 07:02:05.978887 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:05.978859 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-zzn7z_808a129f-fd2d-475c-b70e-41bf5695490b/kube-rbac-proxy/0.log" Oct 30 07:02:06.099349 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:06.099287 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-9qgm5_ba0e8d87-ccf1-4b09-8392-f9d8a02a9444/prometheus-operator-admission-webhook/0.log" Oct 30 07:02:06.611160 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:06.611127 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6577b95894-swx2g_7bfe7f12-7c1e-4e56-8f6d-b94527d0428e/manager/0.log" Oct 30 07:02:06.767101 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:06.767063 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwq2b_a879b8c8-3bc4-4f44-9691-e924c6b703a8/manager/0.log" Oct 30 07:02:06.882921 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:06.882836 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-pjrmv" Oct 30 07:02:10.763061 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:10.763032 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c4bb65c68-jwxlp_d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a/kube-auth-proxy/0.log" Oct 30 07:02:11.157441 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:11.157363 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-f26kj_86cdedee-43d9-421a-9ffd-cfc6775eed66/payload-pre-processing/0.log" Oct 30 07:02:11.220255 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:11.220212 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hpphs_9b2cce36-89ca-4ea5-a488-be5af53dbf14/payload-processing/0.log" Oct 30 07:02:11.254087 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:11.254050 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58849b4f7d-4clbp_38186dd9-9779-4928-a9ae-f566d436bef5/console/0.log" Oct 30 07:02:11.392917 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:11.392884 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-v8lfw_b306ac8e-f3f7-46f2-a46e-444351128037/download-server/0.log" Oct 30 07:02:12.550003 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:12.549957 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c4bb65c68-jwxlp_d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a/kube-auth-proxy/0.log" Oct 30 07:02:12.989458 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:12.989427 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-f26kj_86cdedee-43d9-421a-9ffd-cfc6775eed66/payload-pre-processing/0.log" Oct 30 07:02:13.064999 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:13.064975 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hpphs_9b2cce36-89ca-4ea5-a488-be5af53dbf14/payload-processing/0.log" Oct 30 07:02:15.604961 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:15.604910 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7b2h7_f2c8f0b4-b6a7-4055-abca-b49923505435/dns/0.log" Oct 30 07:02:15.727551 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:15.727511 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7b2h7_f2c8f0b4-b6a7-4055-abca-b49923505435/kube-rbac-proxy/0.log" Oct 30 07:02:16.110785 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:16.110757 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b4v22_099ea3b3-43c1-4bbf-81aa-b3cbd803df66/dns-node-resolver/0.log" Oct 30 07:02:17.493424 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:17.493396 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c4bb65c68-jwxlp_d3a62ddf-f59d-4c0e-bbc5-2c90a8517c6a/kube-auth-proxy/0.log" Oct 30 07:02:17.563318 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:17.563265 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58bb7d9c8f-5xwq2_398c5a0b-aea9-499f-a423-6be32a46ee4d/registry/0.log" Oct 30 07:02:17.830187 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:17.830095 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v2swz_a98d05d0-c37f-4a97-b943-fd4b728e29d0/node-ca/0.log" Oct 30 07:02:17.921739 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:17.921710 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-f26kj_86cdedee-43d9-421a-9ffd-cfc6775eed66/payload-pre-processing/0.log" Oct 30 07:02:17.996743 ip-10-0-136-123 kubenswrapper[4431]: I1030 07:02:17.996706 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hpphs_9b2cce36-89ca-4ea5-a488-be5af53dbf14/payload-processing/0.log"